Segmented layered image system

ABSTRACT

Systems and methods for encoding and decoding document images are disclosed. Document images are segmented into multiple layers according to a mask. The multiple layers are non-binary. The respective layers can then be processed and compressed separately in order to achieve better compression of the document image overall. A mask is generated from a document image. The mask is generated so as to reduce an estimate of compression for the combined size of the mask and multiple layers of the document image. The mask is then employed to segment the document image into the multiple layers. The mask determines or allocates pixels of the document image into respective layers. The mask and the multiple layers are processed and encoded separately so as to improve compression of the document image overall and to improve the speed of so doing. The multiple layers are non-binary images and can, for example, comprise a foreground image and a background image.

This application is a continuation-in-part of U.S. Utility ApplicationSer. No. 10/133,842 which was filed Apr. 25, 2002, entitled ACTIVITYDETECTOR, U.S. Utility Application Ser. No. 10/133,558 which was filedApr. 25, 2002, entitled CLUSTERING, and of U.S. Utility Application Ser.No. 10/133,939 which was filed Apr. 25, 2002, entitled LAYOUT ANALYSTS.This application is also related to co-pending U.S. application Ser. No.10/180,771 entitled SYSTEM AND METHOD FACILITATING DOCUMENT IMAGECOMPRESSION UTILIZING A MASK, the entirety of which is incorporatedherein by reference. This application is also related to co-pending U.S.application Ser. No. 10/180,649 entitled BLOCK RETOUCHING, the entiretyof which is incorporated herein by reference.

TECHNICAL FIELD

The present invention relates generally to document image processing,and more particularly to systems and methods for identifying andcompressing document images.

BACKGROUND OF THE INVENTION

The amount of information available via computers has dramaticallyincreased with the wide spread proliferation of computer networks, theInternet and digital storage means. With such an increased amount ofinformation has come the need to transmit information quickly and tostore the information efficiently. Data compression is a technology thatfacilitates effectively transmitting and storing of information

Data compression reduces an amount of space necessary to representinformation, and can be used for many information types. The demand forcompression of digital information, including images, text, audio andvideo has been ever increasing. Typically, data compression is used withstandard computer systems; however, other technologies make use of datacompression, such as but not limited to digital and satellite televisionas well as cellular/digital phones.

As the demand for handling, transmitting and processing large amounts ofinformation increases, the demand for compression of such data increasesas well. Although storage device capacity has increased significantly,the demand for information has outpaced capacity advancements. Forexample, an uncompressed image can require 5 megabytes of space whereasthe same image can be compressed and require, for example, only 2.5megabytes of space for lossless compression or 500 kilobytes of spacefor lossy compression. Thus, data compression facilitates transferringlarger amounts of information. Even with the increase of transmissionrates, such as broadband, DSL, cable modem Internet and the like,transmission limits are easily reached with uncompressed information.For example, transmission of an uncompressed image over a DSL line cantake ten minutes. However, the same image can be transmitted in aboutone minute when compressed thus providing a ten-fold gain in datathroughput.

In general, there are two types of compression, lossless and lossy.Lossless compression allows exact original data to be recovered aftercompression, while lossy compression allows for data recovered aftercompression to differ from the original data. A tradeoff exists betweenthe two compression modes in that lossy compression provides for abetter compression ratio than lossless compression because some degreeof data integrity compromise is tolerated. Lossless compression may beused, for example, when compressing critical text, because failure toreconstruct exactly the data can dramatically affect quality andreadability of the text. Lossy compression can be used with images ornon-critical text where a certain amount of distortion or noise iseither acceptable or imperceptible to human senses. Data compression isespecially applicable to digital representations of documents (digitaldocuments). Typically, digital documents include text, images and/ortext and images. In addition to using less storage space for currentdigital data, compact storage without significant degradation of qualitywould encourage digitization of current hardcopies of documents makingpaperless offices more feasible. Striving toward such paperless officesis a goal for many businesses because paperless offices providebenefits, such as allowing easy access to information, reducingenvironmental costs, reducing storage costs and the like. Furthermore,decreasing file sizes of digital documents through compression permitsmore efficient use of Internet bandwidth, thus allowing for fastertransmission of more information and a reduction of network congestion.Reducing required storage for information, movement toward efficientpaperless offices, and increasing Internet bandwidth efficiency are justsome of many significant benefits associated with compressiontechnology.

Compression of digital documents should satisfy certain goals in orderto make use of digital documents more attractive. First, the compressionshould enable compressing and decompressing large amounts of informationin a small amount of time. Secondly, the compression should provide foraccurately reproducing the digital document. Additionally, datacompression of digital documents should make use of an intended purposeor ultimate use of a document. Some digital documents are employed forfiling or providing hard copies. Other documents may be revised and/oredited. Many conventional data compression methodologies fail to handlere-flowing of text and/or images when viewed, and fail to provideefficient and effective means to enable compression technology torecognized characters and re-flow them to word processors, personaldigital assistants (PDAs), cellular phones, and the like. Therefore, ifhard copy office documents are scanned into digital form, currentcompression technology can make it difficult if not impossible toupdate, amend, or in general change the digitized document.

Often, compression schemes are tailored to a particular type ofdocument, such as binary, non-binary, textual or image, in order toincrease compression. However, a compression scheme tailored for onetype of document does not typically perform well for other types ofdocuments. For example, a compression scheme tailored for textual baseddocuments does not generally perform well with an image document. Onesolution to this problem is to select a compression scheme tailored tothe type of document or image to be encoded. However, this solution canfail for digital documents which have more than one type of informationin a single document. For example, a digital document can have ahi-color image along with textual information, such as is commonly seenin magazine articles. One approach to overcome this failing is toanalyze a document and divide it into various regions. The variousregions can be analyzed to determine the type of information containedwithin the reasons. A compression scheme can be selected for each regionbased on the type of information. However, this approach can be quitedifficult to implement and requires regions of a variety of sizes andshapes which cause difficulties for compression. Another approach is toseparate a document into a background and a constant color image. Thiscan be helpful because a different compression scheme can be used forthe background and the constant color image. However, the constant colorimage can cause information to be lost by forcing pixel values to be aconstant color.

Additionally, data compression of digital documents should make use ofthe purpose of a document. Some digital documents are used for filing orproviding hard copies. Other documents may be revised and/or edited.Current data compression fails to handle re-flowing of text and/orimages when viewed, and fails to provide efficient and effective meansto enable compression technology to recognized characters and re-flowthem to word processors, personal digital assistants (PDAs), cellularphones, and the like. Therefore, if hard copy office documents arescanned into digital form, current compression technology can make itdifficult if not impossible to update, amend, or in general change thedigitized document.

SUMMARY OF THE INVENTION

The following is a summary of the invention in order to provide a basicunderstanding of some aspects of the invention. This summary is notintended to identify key/critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentsome concepts of the invention in a simplified form as a prelude to themore detailed description that is presented later.

The present invention relates generally to systems and methods forencoding and decoding document images. Document images are segmentedinto multiple layers according to a mask, where the multiple layers arerepresented as non-binary. The respective layers can then be processedand compressed separately in order to facilitate better compression ofthe document image overall.

According to one aspect of the invention, a mask is generated from adocument image. The mask is generated so as to reduce an estimate ofcompression for the combined size of the mask and multiple layers of thedocument image. The mask is then employed to segment the document imageinto the multiple layers. The mask determines or allocates pixels of thedocument image into respective layers. The mask and the multiple layersare processed and encoded separately so as to improve compression of thedocument image overall and to improve the speed of so doing. Themultiple layers are non-binary images and can, for example, comprise aforeground image and a background image.

According to another aspect of the invention, a document image issegmented into multiple layers, the multiple layers comprising aforeground image, a background image and a mask. The mask is a binaryimage and used to segment the document image into the foreground imageand the background image. Generally, the mask is generated so as toreduce an estimate of a combined size of the mask, the foreground imageand the background image. Unlike some conventional systems that limit alayer to a single color, both the foreground image and the backgroundimage can use any suitable range of colors and are not limited to asingle or constant color. Additional processing of the mask, theforeground image and the background image can be performed. The mask,the foreground image and the background image are encoded and combinedinto a single bitstream. Any number of suitable compression schemes canbe used for encoding purposes. For decoding, a compressed bitstream isseparated into a mask bitstream, a foreground bitstream and a backgroundbitstream. The mask bitstream, the foreground bitstream and thebackground bitstream are decoded into a mask, a foreground image and abackground image. Additional processing of the mask, the foregroundimage and the background image can be performed. The foreground imageand the background image are combined into a recombined document imageaccording to the mask.

According to yet another aspect of the invention, a mask separatorreceives a document image and generates a mask from the document image.The mask is represented in binary format. A foreground backgroundsegmenter receives the mask and the document image and segments thedocument image into a foreground image and a background image. A maskencoder encodes the mask into a mask bitstream. A foreground encoderencodes the foreground image into a foreground bitstream. A backgroundencoder encodes the background into a background bitstream. A combinercomponent combines the mask bitstream, the foreground bitstream and thebackground bitstream into a combined bitstream.

To the accomplishment of the foregoing and related ends, certainillustrative aspects of the invention are described herein in connectionwith the following description and the annexed drawings. These aspectsare indicative of various ways in which the invention may be practiced,all of which are intended to be covered by the present invention. Otheradvantages and novel features of the invention may become apparent fromthe following detailed description of the invention when considered inconjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a segmented layered image encoding systemaccording to one aspect of the invention.

FIG. 2 is a diagram of one exemplary step of a wavelet computation.

FIG. 3 is a diagram illustrating an exemplary step of a waveletcomputation where some pixels are missing.

FIG. 4 is a diagram illustrating an exemplary step of a masked waveletcomputation.

FIG. 5 is a block diagram of a segmented layered image decoding systemaccording to one aspect of the invention.

FIG. 6 illustrates a sample document image according to one aspect ofthe present invention.

FIG. 7 is a block diagram of a segmented layered image encoding systemaccording to one aspect of the invention.

FIG. 8 is a block diagram of a segmented layered image decoding systemaccording to one aspect of the invention.

FIG. 9 is a flow diagram of a method of encoding a document according toone aspect of the invention.

FIG. 10 is a flow diagram of a method of encoding a document accordingto one aspect of the invention.

FIG. 11 is a schematic block diagram of an exemplary operatingenvironment for a system configured in accordance with the presentinvention.

FIG. 12 is a schematic block diagram of an exemplary communicationenvironment in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. It may be evident, however, thatthe present invention may be practiced without these specific details.In other instances, well-known structures and devices are shown in blockdiagram form in order to facilitate describing the present invention.

As used in this application, the term “component” is intended to referto a computer-related entity, either hardware, a combination of hardwareand software, software, or software in execution. For example, acomponent can be, but is not limited to being, a process running on aprocessor, a processor, an object, an executable, a thread of execution,a program, and a computer. By way of illustration, both an applicationrunning on a server and the server can be a component. One or morecomponents can reside within a process and/or thread of execution and acomponent may be localized on one computer and/or distributed betweentwo or more computers.

Further, “document image” is intended to refer to a digitalrepresentation of document(s) comprising one or more color(s) (e.g.,binary (black/white), gray-scale and/or color document(s)).Additionally, a document image can have image(s), text and/or text withimages, with potential superimposition of text and images. A documentimage can comprise binary, RGB, YUV and/or other representations ofdocument(s). An RGB document image is represented red, green and bluecomponents. A YUV document image is represented using a luminescencecomponent denoted by Y and chrominance components denoted by U and V.The YUV representation is, generally, more suitable for compressionbecause the human eye is less sensitive to U and V distortion, andtherefore U and V can be subsampled by a factor 2, and because Ycaptures the correlations between R, G and B. For the purpose of textclustering, the Y representation is particularly interesting becausetext is much easier to read when resulting from a change of luminance.Text resulting from a change of chrominance, for instance from red togreen, at a given luminance, is much harder to read. A color document,can therefore be transformed into a YUV document, which can then bebinarized, without much loss of textual information. A document imagecomprises picture elements commonly referred to as “pixels”. A documentimage can be based on single or multi-page document(s) of any shape orsize.

FIG. 1 is a high-level block diagram of a segmented layered imageencoding system 100 according to one aspect of the invention. The system100 identifies and compresses text, handwriting, drawings and the like,and the system can operate on single or multi-page documents of anysuitable shape or size. The system 100 includes a mask separator 102, aforeground background segmenter 104, a mask encoder 106, a foregroundencoder 108, a background encoder 110 and a combine component 112. Themask separator 102 receives a document image (e.g., digitalrepresentation of a document) and generates a mask. The document imagecan have one or more pages, and is typically scanned from a document.The document image can have any resolution, which is generally expressedas dots per inch (dpi). For example, faxed documents typically use aresolution of about 150–200 dpi. Additionally, the document image canhave substantially any pixel size or document size, for example, 640pixels by 480 pixels and/or A4 size.

The mask generated by the mask separator 102 can be employed to segmentthe document image into two layers, a foreground image and a backgroundimage. It is appreciated that alternate aspects of the invention cansegment the image into more than two layers. The mask, also referred toas the mask image, is a binary image, where a pixel value determineswhether that respective pixel belongs in the foreground image or thebackground image. The mask separator 102 generates the mask such that acombined size of the foreground image and the background image isreduced. It is appreciated that this concept can be extended to multiplemasks and multiple foregrounds and still be in accordance with thepresent invention.

A number of approaches can be used to generate the mask. Forcompression, like pixel values compress better than differing pixelvalues. For example, a region of blue sky compresses better than aregion of varying colors and intensities. As stated above, the mask isgenerated to reduce the combined size of the mask, the foreground imageand the background image.

One approach that can be used is to generate a superset of possiblemasks and select a best mask of the superset. For a document imagehaving N pixels, there are 2^(N) possible masks. Thus, it is possible toanalyze every possible mask and determine which mask generates asmallest overall combined image. However, analyzing every possible maskcan be computationally expensive, time consuming and generally is notpracticable.

Another approach is to subdivide the document into regions, analyze eachregion to allocate pixels and merge regions based at least in part uponenergy (e.g., energy variance). An energy variance (which is an estimateof compression that would be obtained) is a measure based on a sum ofthe square of distances. Small regions of the document, for example 2×2pixels or 4×4 pixels, can be analyzed for energy variance. The smallregions can be segmented into foreground and background regions suchthat the energy variance of that small region is reduced or minimized.Every possible mask can be used to determine which mask to use becausethe analyzed region is small enough to make such analysis feasible. Forexample, a 4 pixel region has only 16 possible permutations of a maskfor that region. The small regions can then be merged (e.g., by pairs)together to generate the mask, overall, that reduces the combined sizeof the mask, the foreground image and the background image.

The foreground background segmenter 104 receives the mask from the maskseparator 102 and the document image. The foreground backgroundsegmenter 104 uses the mask to segment the document image into theforeground image and the background image. For each pixel of thedocument image, a corresponding pixel of the mask is referenced. Thepixel is allocated to the foreground image or the background image basedon the corresponding pixel of the mask. For example, if thecorresponding pixel of the mask is a “1”, the pixel is assigned to theforeground image. Conversely, if the corresponding pixel of the mask isa “0”, the pixel is assigned to the background image. It is appreciatedhowever that whether “0” or “1” indicates foreground or background canbe user-defined (e.g., “1” assigned to background and “0” assigned toforeground).

Additionally, the foreground image and the background image are disjointof each other. However, additional processing of the mask can beperformed to yield different foreground and background masks, which cansometimes overlap, to decrease artifacts along mask boundaries, orignore certain pixels entirely, to increase compression. The foregroundmask is then used to segment the foreground image from the documentimage and the background mask is used to segment the background imagefrom the document image. In this case, the foreground image and thebackground image are not disjoint of each other but overlapslightly—such overlap can decrease compression but reduce edge effects.Even if the foreground mask and the background mask are employed tosegment the document image, the foreground mask and the background maskare not required after the document image is segmented.

The background image and the foreground image have holes or vacantregions where pixels were assigned to the other image. These vacantregions can also be referred to as don't care pixels. In contrast, thenon-vacant regions can be referred to as do care pixels. The vacantregions can be handled in any suitable manner so as to reduce overallcompression size of the foreground and background images. One approachis to fill these vacant regions with don't care pixel values. The don'tcare pixel values are selected so as to increase compression andthereby, reduce size of the image. For example, an exemplary backgroundimage can be white but have don't care pixels where text was located.The don't care pixels, in this example, can be filled with white toimprove compression. Another approach is to not fill the vacant regionsand utilize a compression scheme that takes advantage of the vacantregions, such as for example masked wavelet compression. Otherapproaches can be used in accordance with the invention.

Additional processing can also be performed on the background image andthe foreground image. The background and foreground images can bedownsized to a lower resolution, thus reducing respective compressedsize. For example, the background and foreground images can be downsizedfrom 300 dpi to 100 dpi. Also, constant color connected components canbe identified in the background and foreground images. Constant colorconnect components are a plurality of connected pixels havingsubstantially the same color values. The constant color connectedcomponents can be identified by locating a seed pixel and utilizing analgorithm to determine substantially same color pixels connectedthereto. A threshold value or minimum pixel can be set so that theconstant color connected components have a minimum number of pixels. Theidentified constant color connected components can increase compressionof the document image.

The mask encoder 106 receives the mask from the mask separator 102 andencodes the mask to generate compressed bits or a compressed mask streamof bits from the mask. Any suitable compression scheme can be used forthe mask encoder 106. The mask is binary and typically includes textualinformation. Thus, a compression scheme should be select that compressesbinary and textual information well. A bi-level compression scheme canbe used to encode the mask.

A compression scheme that can be used to encode the mask is CCITT(Consultative Committee on International Telephone and Telegraph). TheCCITT is currently known as ITU-T International TelecommunicationsUnion-Telecommunications Sector (changed name in 1994)) which is astandards group and a name of a lossless compression technology forfax/modem communication. This type of compression works well with binaryimages. Typical compression ratios are 4:1 for the older versionV.42bis, and 6:1 for newer version V.44 2000, which is based on theLempel-Ziv-Jeff-Heath (LZJH) compression algorithm. It is to beappreciated that other suitable compression methods or schemes can beused to encode the mask in accordance with the present invention.

The foreground encoder 108 receives the foreground image from theforeground background segmenter 104 and encodes the foreground imageinto a foreground bitstream. The background encoder 110 receives thebackground image from the foreground background segmenter 104 andencodes the background image into a background bitstream. Any suitablecompression scheme can be used for the foreground encoder 108 and thebackground encoder 110. For example, progressive wavelength encoding(PWC), progressive transform coding (PTC), JPEG, JPEG 2000, and maskedwavelet compression schemes can be used to encode the foreground imageor background region. For some compression schemes (e.g., maskedwavelet), the foreground encoder 108 and the background encoder 110 canrequire the mask in order to encode the foreground image and thebackground image.

The combiner component 112 receives the compressed bits from the maskencoder 106, the foreground encoder 108 and the background encoder 110and combines the bits into an output stream or output file. The combinercomponent 112 can include header information in the output fileidentifying or providing information such as encoding types,dictionaries and the like that can be used by a decoder to reconstructthe document image.

For illustrative purposes, the encoding system has been described asprocessing the entire image at once via the various components describedsupra. However, it is appreciated that overlapping scan regions can beused to reduce memory usage. For example, a first strip of 512 lines ofthe document image can be processed by the system. Then, a second stripoverlapping the first strip by an overlap amount, such as 10 lines, isprocessed by the system. Subsequent strips are processed until theentire document image has been processed—the overlap amount facilitatesmitigating variations between strips.

As stated above, the foreground image and the background image havevacant or don't care regions. The vacant regions can be handled a numberof ways. One approach is to fill the vacant regions with data and thenuse a regular compression technique. A simple process to fill the vacantregions of the images is to fill those vacant regions with an averagepixel value for that image. However, this process can create sharpdiscontinuities at the mask boundaries, can increase a required bit ratefor a given peak signal-to-noise ratio (PSNR) and produce noticeableringing near the mask or vacant region boundaries. Another process is tocolor each pixel with the color of the closest non-masked (or do careregion) pixel. A standard morphology algorithm allows that process to beperformed with only two passes over all the pixels leading toVoronoi-filled vacant regions under the mask. Then, when the foregroundor background image is reconstructed, the reconstructed image islow-passed and then the known pixels are restored to their correctvalues. If the low-pass filter cutoff frequency is too low, sharp edgescan occur causing an increase in the required bit rate and noticeableringing near boundaries.

Another approach to handling vacant regions is to use projection ontoconvex sets. For example, consider two convex sets: the set of imagesthat matches the input on the visible pixels, and the set of images thathave certain wavelet coefficients set to zero (e.g. all high-frequencycoefficients beyond a certain resolution level). By alternatingprojection onto those two sets, an image can be found that agrees withthe visible pixels and which compresses well because of the many zerowavelet coefficients.

Yet another approach to addressing vacant regions is to employ wavelettransforms for the foreground encoder 108 and the background encoder 110designed explicitly for irregular grids. Such wavelet decompositions areemployed in connection with computer vision and compression of geometrydata in computer graphics for example. Such wavelets are adapted to theirregular pattern of the mask.

A wavelet transform compression scheme that can be employed is a maskedwavelet transform. The masked wavelet transform can be used for theforeground encoder 108 and the background encoder 110 where the waveletfunction changes on a case-by-case basis as a function of the mask. Itis appreciated that the mask is provided to the foreground encoder 108and the background encoder, although not shown in FIG. 1, in order touse the masked wavelet transform compression scheme. For example, aregular conventional wavelet function may use k values for theprediction. However, with the mask and vacant regions, the regularwavelet function cannot handle pixels with no values, (e.g., don't carepixels in the don't care regions or vacant regions). In contrast, thechangeable wavelet function employs only available values or visibleregions of the foreground and the background images. Thus, the maskedwavelet transform avoids the need to fill vacant regions with don't carepixels.

FIG. 2 is a diagram of one step of a traditional computation of awavelet using “lifting”. The diagram illustrates, for a cubic wavelet, aprediction step at position 3, and a corresponding update step atposition 6 (the other positions are omitted in the diagram for clarity).The coefficient next to each arrow indicates how to compute a linearcombination in order to perform each step. For instance, a “detail”value (resulting from the high pass filter) at position 3 is computed bycomputing the following equation:d ₃ =s ₃−(−s ₀+9s ₂+9s ₄ −s ₆)/16  Eq. 1

The update step is computed via the following equation:s ₆ =d ₆+(−d ₃+9d ₅+9d ₇ −d ₉)/32  Eq. 2

FIG. 3 illustrates a problem that arises when some pixel values aremissing. In the diagram, no value is available for positions 1, 2, 5, 8,and 9. Obviously, computing a traditional cubic wavelet would not workbecause the result would depend on missing values. Setting the missingvalues to some constant (zero, or some average over the entire image)can introduce sharp discontinuities, which translate into poorcompression and/or undesirable artifacts. In contrast, the maskedwavelet compression scheme alters the wavelet function on a case-by-casebasis as a function of the mask. For instance, during the predictionstep of lifting, if k values are available for prediction, a polynomialof degree k−1 is used for interpolation. When only three values areavailable, a quadratic instead of cubic polynomial is employed. If onlyone value was available, a constant polynomial would be used, and thewavelet would be a Haar wavelet. Note that if the signal is a polynomialof degree k−1 and k pixels are not masked, the prediction is perfectlyaccurate.

The masked wavelet transform compression scheme utilizes computation ofa prediction for all the coefficients at odd positions from thecoefficients at even positions. The difference between the oddcoefficient and its prediction is the wavelet coefficient—this can beviewed as a high pass filter with some zeros at the odd positions. Asignal s is centered on a position and indexed accordingly. Forsimplicity, the scheme is described with respect to a 7-tap filter, asshown in FIG. 4. FIG. 4 illustrates a lifting step where s₀ is predictedas a function of s⁻³,s⁻¹,s₁,s₃ and the residual d is computed.

A filter a is indexed from 0 to i in accordance to the standard matrixnotation. The wavelet coefficient d is provided by equation:

$\begin{matrix}{d = {s_{0} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{s_{{2i} - {k/2}}a_{i}}}}} & {{Eq}.\mspace{14mu} 3}\end{matrix}$where k is a number of taps in the filter (in this case k=7). Themoments M of the high pass filter can be written as (settings_(i)=i^(n)):

$\begin{matrix}{M_{n} = {0^{n} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{\left( {{2i} - {k/2}} \right)^{''}a_{i}}}}} & {{Eq}.\mspace{14mu} 4}\end{matrix}$

Assuming that a regular signal can be approximated by a low orderpolynomial (using Taylor expansion) of order j, then a is chosen so asto set the first j+1 moment to zero. Then the wavelet transform willhave many zeros and compress well. Thus, if k=7, a has 4 degrees offreedom and the first 4 moments are set to zero. These results areverified in the following system:

$\begin{matrix}{{\begin{bmatrix}1 & 1 & 1 & 1 \\{- 3} & {- 1} & 1 & 3 \\9 & 1 & 1 & 9 \\{- 27} & {- 1} & 1 & 27\end{bmatrix}\begin{bmatrix}a_{0} \\a_{1} \\a_{2} \\a_{3}\end{bmatrix}} = \begin{bmatrix}{- 1} \\0 \\0 \\0\end{bmatrix}} & {{Eq}.\mspace{14mu} 5}\end{matrix}$which can be written in matrix notation as: wa=c. The solution to thissystem is the coefficients used in the well know cubic wavelet:a=[1/16,−9/16,−9/16,1/16].

The system is generalized to the case when a mask is present, such asthe mask from FIG. 1. In other words, some of the coefficientss_(2i−k/2) are missing, which can be modeled by introducing a matrix m,

$\begin{matrix}{m = \begin{bmatrix}m_{0} & 0 & 0 & 0 \\0 & m_{1} & 0 & 0 \\0 & 0 & m_{2} & 0 \\0 & 0 & 0 & m_{3}\end{bmatrix}} & {{Eq}.\mspace{14mu} 6}\end{matrix}$where m_(i)ε{0,1}, such that: wma=c. The effect of m is to remove columnin the system of Eq. 5. Unfortunately, in this system, a_(i) isunder-determined when m_(i)=0, and over-determined otherwise. The reasonis that there are too many vanishing moments constraints imposed on thea_(i) corresponding to visible pixels. This can be fixed by imposingthat only first j moments should be 0, if there are j coefficients suchthat m_(i)≠0. This corresponds to retaining only the first j equationsof the system in Eq. 5. For the under-determined coefficients a_(i)(when a_(i) is multiplied by m_(i)=0), an arbitrary constraint a_(i)=0is added; and such can be accomplished in a single system:(pwm+m−I)a=pc  Eq. 7where I is the identity matrix, and:

$\begin{matrix}{p = \begin{bmatrix}p_{0} & 0 & 0 & 0 \\0 & p_{1} & 0 & 0 \\0 & 0 & p_{2} & 0 \\0 & 0 & 0 & p_{3}\end{bmatrix}} & {{Eq}.\mspace{14mu} 8}\end{matrix}$withp _(i)=(tr(m)>i)?1:0;  Eq. 9

The constraints on p_(i) ensure that lines in the system of Eq. 5 areremoved from the bottom coefficients of the system of Eq. 5 for each 0coefficient in m. In other words, if there are j coefficients m_(i)which are not equal to 0, then p_(i)=1 for i=[0 . . . j−1], and p_(i)=0otherwise. It can easily be verified that the system always has a uniquesolution for values of m_(i)ε{0,1}. The solutions for a for everypossible values of m are provided by system Eq. 9 and summarized for thecubic wavelet in TABLE 1., system is 4.

TABLE 1 (m₀, m₁, m₂, m₃) a₀ a₁ a₂ a₃ 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 −1  0 0 1 0 0 0 −1   0 0 0 1 1 0 0 −3/2 1/2 0 1 0 0 0 −1   0 0 0 1 0 1 0−3/4 0 −1/4   0 1 1 0 0 −1/2 −1/2 0 0 1 1 1 0 −3/8 −3/4 1/8 1 0 0 0 −1  0 0 0 1 0 0 1 −1/2   0 0 −1/2   1 0 1 0 −1/4   0 −3/4 0 1 0 1 1 −1/8   0−9/8   1/4 1 1 0 0 1/2 −3/2 0 0 1 1 0 1 1/4 −9/8 0 −1/8 1 1 1 0 1/8−3/4   −3/8   0 1 1 1 1  1/16 −9/16 −9/16  1/16

Next, the signal s is updated with the predictions from above. In theupdate step, it is desirable for the first moments of the low passfilter to vanish, after the signal has been multiplied by (−1)^(i). Inother words, if a regular signal (e.g., can be written as a lower orderpolynomial) is multiplied by the highest frequency signal, (−1)^(i),then the low pass filter should output zero. This condition can easilybe cast as 0-moment constraint, as in the previous section, except thatthe input will be of the form s_(i)=(−1)^(i)i^(n) instead ofs_(i)=i^(n). Using similar notation as for the predict step, the updatestep corresponds to the equation:

$\begin{matrix}{s = {s_{0} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{d_{{2i} - {k/2}}b_{i}}}}} & {{Eq}.\mspace{14mu} 10}\end{matrix}$Where k is the number of tap in the filter (in this case k=7). Themoments of the low pass filter can be written as

$\begin{matrix}{M_{n} = {0^{n} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{d_{{2i} - {k/2}}b_{i}}}}} & {{Eq}.\mspace{14mu} 11}\end{matrix}$but, for each d_(i), the equation can be rewritten locally as (assumings_(i)=(−1)^(i)i^(n)):

$\begin{matrix}{d_{j} = {{s_{j} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{s_{j + {2i} - {k/2}}a_{i}}}} = {{- j^{n}} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{\left( {j + {2i} - {k/2}} \right)^{n}a_{i}}}}}} & {{Eq}.\mspace{14mu} 12}\end{matrix}$since j and k/2 are odd. Because individual a_(i) are set to generatezero moments, the following is true:

$\begin{matrix}{0 = {j^{n} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{\left( {j + {2i} - {k/2}} \right)^{n}a_{i}}}}} & {{Eq}.\mspace{14mu} 13}\end{matrix}$which implies d_(j)=−2j^(n). Thus:

$\begin{matrix}{M_{n} = {0^{n} + {\sum\limits_{i = 0}^{i = {|{k/2}|}}{{- 2}\left( {{2i} - {k/2}} \right)^{n}b_{i}}}}} & {{Eq}.\mspace{14mu} 14}\end{matrix}$

For the wavelet to compress as well as a regular signal, as many momentsas possible should be equal to zero as possible. Since there are 4degrees of freedom, the first 4 moments can be set to zero. It is easyto verify this results in the following system:

$\begin{matrix}{{\begin{bmatrix}1 & 1 & 1 & 1 \\{- 3} & {- 1} & 1 & 3 \\9 & 1 & 1 & 9 \\{- 27} & {- 1} & 1 & 27\end{bmatrix}\begin{bmatrix}b_{0} \\b_{1} \\b_{2} \\b_{3}\end{bmatrix}} = \begin{bmatrix}{1/2} \\0 \\0 \\0\end{bmatrix}} & {{Eq}.\mspace{14mu} 15}\end{matrix}$Which can be written in matrix notation as wb=c′. The solution to thissystem are the coefficients used in the well know cubic wavelet:b=[−1/32,9/32,9/32,−1/32].

Now, it is assumed that some of the coefficients s_(i) are missing.First, it is assumed that all missing values are only at even locations.This system can be solved in a similar fashion as before to verify that:(pwm+m−I)b=pc′  Eq. 16Note that m and p matrix depend on the location centered in s₀. Eachlocation views a different part of the mask and has therefore its own mand p. Solutions to Eq. 16 are provided in TABLE 2, illustrated below,which depict solutions of equation Eq. 16 for different values of m whenthe dimension of the system is 4. To derive Eq. 16, the odd locationswere assumed not masked. If some odd locations are masked, but thenumber of masked value is less than n, then d_(j)=−2j^(n) and Eq. 16holds. Otherwise, there are two many masked pixels in the predict stepto nullify the n-th moment in the update step (the wavelet coefficientwill still be as small, but not zero). The inverse wavelet transform iseasily computed by undoing each step locally, thanks to the liftingformalism.

TABLE 2 (m₀, m₁, m₂, m₃) b₀ b₁ b₂ b₃ 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1/2 00 1 0 0 0 1/2 0 0 0 1 1 0 0 3/4 −1/4   0 1 0 0 0 1/2 0 0 0 1 0 1 0 3/8 01/8 0 1 1 0 0 1/4 1/4 0 0 1 1 1 0  3/16 3/8 −1/16 1 0 0 0 1/2 0 0 0 1 00 1 1/4 0 0 1/4 1 0 1 0 1/8 0 3/8 0 1 0 1 1  1/16 0  9/16 −1/8   1 1 0 0−1/4   3/4 0 0 1 1 0 1 −1/8    9/16 0  1/16 1 1 1 0 −1/16 3/8  3/16 0 11 1 1 −1/32  9/32  9/32 −1/32

Thus, the masked wavelet transform can be employed to encode theforeground image and the background image without filling the vacantregions. However, it is appreciated that varying implementations canemploy other encoding methods or compression schemes which fill thevacant regions with don't care values as discussed above.

FIG. 5 is a block diagram of a segmented layered image decoding systemaccording to one aspect of the invention. The system receives acompressed bitstream and generates a recombined image from thecompressed bitstream. The system includes a separator component 502, aforeground decoder 504, a background decoder 506, a mask decoder 508 anda combiner 510.

The separator component 502 receives the compressed bits and separatesthe bits to form a foreground bitstream, background bitstream and maskbitstream. The separator component 502 can utilize header information toseparate the bitstreams. The foreground decoder 504 decompresses theforeground bitstream to yield a foreground image. The background decoder506 decompresses the background bitstream to yield a background image.The foreground decoder 504 and the background decoder 506 can use anynumber of decompression schemes such as, for example, progressivewaveform or progressive transform.

The mask decoder 508 decompresses the mask bitstream to yield a mask ormask image. The mask decoder 508 can also retrieve layout and/or re-flowinformation. Additionally, the mask decoder 508 can obtain a dictionaryused for reconstructing textual information. The mask decoder 508typically utilizes a bi-level decompression scheme. For some compressionschemes (e.g., masked wavelet), the foreground decoder 504 and thebackground decoder 506 can require the decompressed mask from the maskdecoder 508 in order to obtain the foreground image and the backgroundimage.

The combiner 510 combines the foreground image, the background image andthe mask into a recombined document image. The recombined image isidentical to or an approximation to an original document image.Additionally, the recombined image can have a different shape and/orsize depending on a desired display. For example, a scan of aletter-sized document could be displayed differently on a portabledevice having a much smaller screen. The combiner 510 generates pixelsfor the recombined document image from the foreground and backgroundimages based on the mask. Each pixel of the recombined document image isdetermined by referencing a corresponding pixel of the mask therebydetermining whether the pixel of the recombined document should comefrom the foreground image or the background image. For example, a maskpixel value of 1 can indicate that the pixel of the recombined documentshould come from the corresponding pixel of the foreground image.

FIG. 6 illustrates sample document images according to an aspect of thepresent invention. FIG. 6 is exemplary and is intended only to furtherillustrate the present invention. A document image 601 is shown with avariety of colors and/or shades. The document image 601 is a typicaldocument image that includes textual information 602 and imageinformation 604 and 603 and a background 605. The patterns shown in FIG.6 illustrate different shades or colors. Thus, the textual information602 can be of a different color than the image information 603 and 604or the background 605. Additionally, the information can use anysuitable number of colors. According the present invention, such asusing the system of FIG. 1, the document image 601 is segmented into 3components, a mask 610, a foreground image 611 and a background image612. Other aspects of the invention can segment a document image intomore than two or three layers.

The mask 610 is a binary image and, thus is shown only with black andwhite regions. The mask 610 determines or allocates whether pixels ofthe document image 601 go to the foreground image 611 or the backgroundimage 612. The mask 610 is also employed to generate a reconstructeddocument image by determining which pixels of the foreground image 611and the background image 612 are in the reconstructed document. Thereconstructed document is identical or approximates the document image601.

As can be seen in FIG. 6, the foreground image 611 includes portions ofthe document image 601 as indicated in the mask 610. Black pixels ofthis mask 610, allocate pixels of the document image 601 to theforeground image 611. Other portions of the foreground image 603 can befilled with don't care pixels 615. As stated above, the don't carepixels 615 increase compression but are not present in the reconstructedimage. Similarly, the background image 612 includes portions of thedocument image 601 as indicated by the mask 610. White pixels of thismask 610 allocate pixels of the document image 601 to the backgroundimage 612. Other portions of the background image 604 can be filled withdon't care pixels 606 as shown in FIG. 6. The don't care pixels 606 arepresent for compression purposes and are not included in thereconstructed image. An outline of the textual information 602 is shownin the background image 612 for illustrative purposes. However, it isappreciated that the textual information 602 can be filled with don'tcare pixels matching the background 605 such that the textualinformation 602 is not visible in the background image 612.

As is stated above, FIG. 6 is but one example of many possible documentimages according to the present invention. Many variations of the imagescan be made and still be in accordance with the present invention.

FIG. 7 is a block diagram of a segmented layered image encoding systemaccording to one aspect of the invention. The system can identifyinformation such as text, handwriting, drawings and the like in adocument and compress the document. The system operates on single ormulti-page documents of any suitable shape or size.

A colorspace converter 702 operates on a document image to convert acolorspace of the document image. The colorspace refers to how colorsare represented in the document image. The colorspace generally has anumber of components that provide a color on a pixel-by-pixel basis. Forexample, an image can be represented with an RGB (red, green and blue)colorspace where each pixel has a value for each of the RGB components.Additionally, a transparency component can be included to indicate atransparency of the pixel. The overall color of the pixel is obtained bythe combination of these components. For example, white can be obtainedby using RGB values of 255, 255 and 255 (assuming 8 bits per component)and blue can be obtained by RGB values of 0, 0 and 200. The RGBcolorspace is often employed for displaying images and/or scanningdocuments. However, the RGB colorspace can hinder compression.

Colors in a document are, generally, correlated and/or interrelated. Foran RGB colorspace, the components are compressed individually (bychannel) which results in the same information being coded multipletimes. Compression of the document image can be improved by convertingthe colorspace of the document to a more compressible colorspace.

One possible colorspace conversion is to convert the document image fromthe RGB colorspace to a YUV colorspace. The Y, U and V components areluminescence, chrominance red and chrominance blue components,respectively. YUV was originally developed for television viewing. Theconversion was developed based on viewing preferences, notcompressibility. Thus, the document image can be converted from the RGBcolorspace to the YUV colorspace using the following equation:Y=0.6G+0.3R+0.1BU=R−Y  Eq. 17V=B−Y

The YUV colorspace does,provide better compression compared to the RGBcolorspace because it utilizes correlation between the components sothat the same information isn't coded multiple times. Most of thecorrelation information is contained in the Y component. Eq. 17illustrates that the YUV components can be represented by smaller valuesthan corresponding RGB components. However, as stated above, the YUVcolorspace was not designed for compressibility, but more forviewability. It is appreciated that the document image can be convertedback to RGB colorspace for any suitable purpose, such as for exampledisplaying, by modifying Eq. 17.

Another colorspace conversion is to convert the document image from theRGB colorspace to an YC_(o)C_(g) colorspace. The YC_(o)C_(g)representation utilizes luminescence represented by Y,chrominance-orange represented by C_(o) and chrominance-greenrepresented by C_(g). The RGB components can be mapped to YC_(o)C_(g)(e.g., as an alternative to the conventional Y U V described above)utilizing the transform.

$\begin{matrix}{\begin{bmatrix}Y \\C_{o} \\C_{g}\end{bmatrix} = {\left. {\begin{bmatrix}1 & 2 & 1 \\2 & 0 & {- 2} \\{- 1} & 2 & {- 1}\end{bmatrix}\begin{bmatrix}R \\G \\B\end{bmatrix}}\Leftrightarrow\begin{bmatrix}R \\G \\B\end{bmatrix} \right. = {\begin{bmatrix}1 & 1 & {- 1} \\1 & 0 & 1 \\1 & {- 1} & {- 1}\end{bmatrix}\begin{bmatrix}Y \\C_{o} \\C_{g}\end{bmatrix}}}} & {{Eq}.\mspace{14mu} 18}\end{matrix}$Significantly, an advantage of YCoCg color space mapping is that fromRGB to YCoCg and the inverse conversion from YCoCg to RGB can beaccomplished utilizing integer arithmetic. Further, the inverseconversion can be performed without multiplication. The YC_(o)C_(g)color space representation can result in significantly bettercompression performance than the popular YUV because it is a betterapproximation to the statistically optimal spaces that are obtained froma principal component analysis on modern digital picture data. Theadvanced YC_(o)C_(g) colorspace is not designed for viewing purposes,but for compression purposes. Furthermore, since the equations ortransform of Eq. 18 utilizes integer arithmetic, the document image canbe converted to the YC_(o)C_(g) colorspace in a computationallyefficient manner.

It is appreciated that the colorspace converter 702 can convert thedocument image from any colorspace to any colorspace. For example, thecolorspace converter 702 could convert a document image from a YUVcolorspace to an YCoCg colorspace. Additionally, if the document imageis already in the preferred colorspace, no conversion is performed.

The mask separator 704 receives a document image from the color spacecomponent 702 and generates a mask. The mask generated by the maskseparator 704 is employed to segment the document image into two layers,a foreground image and a background image. It is appreciated thatalternate aspects of the invention can segment the image into more thantwo layers. The mask, also referred to as the mask image, is a binaryimage, where a value of each pixel determines whether that pixel belongsin the foreground image or the background image. The mask separator 704generates the mask that reduces a combined size of the mask, theforeground image and the background image.

A number of approaches can be employed to generate the mask. Forcompression, similar pixel values compress better than dissimilar pixelvalues. For example, a region of blue sky compresses better than aregion of varying colors and intensities. As stated above, the mask isgenerated to reduce the combined size of the mask, the foreground imageand the background image.

One approach that can be employed is to generate all possible masks andsimply pick the best one. For a document image having N pixels, thereare 2^(N) possible masks. Thus, it is possible to go through everypossible mask and determine which one generates the smallest overallcombined images. However, going through every possible mask iscomputationally expensive, time consuming and generally, not feasible.

Another approach is to subdivide the document into regions, analyze eachregion to allocate pixels and merge the regions. An energy variance,which is an estimate of compression that would be obtained, is a measurebased on a sum of the square of distances. Small regions of thedocument, for example 2×2 pixels or 4×4 pixels, can be analyzed forenergy variance. The small regions can be segmented into foreground andbackground regions such that the energy variance of that small region isreduced or minimized. Every possible mask can be analyzed to determinewhich mask to employ because the region is small enough that suchanalysis is practicable. For example, a 4 pixel region has only 16possible permutations of a mask for that region. The small regions canthen be merged together to generate the mask, overall, that reduces thecombined size of the mask, the foreground image and the backgroundimage.

Another approach to generating a mask assumes that the foreground andthe background are constant over a region, and a mask is generated whichminimizes or substantially reduces the variance or energy variancearound those constants. The variance here is used as an estimate of thecompression obtainable for both the foreground and the background,instead of actually generating the mask, segmenting the document imageto get the foreground and background image and compressing them. Theenergy variance, as discussed above, which is also an energy measure (asum of the square distances), is an acceptable estimate of the size ofthe foreground and background after compression. First, the cost ofcompressing the mask is completely ignored. Further steps can take thesize of the mask into account.

Assume for a given region, that the region is a set S of N pixels, andthat F and B are a partition S (i.e. F∪B=S and F∩B=Ø). If ƒ(x) is theimage value at pixel location x,x∈S, the variance of the foreground andbackground are respectively:

$\begin{matrix}{v_{F} = {\sum\limits_{x\;\varepsilon\; F}\left( {{f(x)} - \mu_{F}} \right)^{2}}} & {{Eq}.\mspace{14mu} 19} \\{v_{B} = {\sum\limits_{x\;\varepsilon\; B}\left( {{f(x)} - \mu_{B}} \right)^{2}}} & {{Eq}.\mspace{14mu} 20}\end{matrix}$Where

$\mu_{F} = {{\frac{1}{N_{F}}{\sum\limits_{x\;\varepsilon\; F}{{f(x)}\mspace{14mu}{and}\mspace{14mu}\mu_{B}}}} = {\frac{1}{N_{B}}{\sum\limits_{x\;\varepsilon\; B}{f(x)}}}}$are respectively the mean of the foreground and the background, andN_(F) and N_(B) are respectively the number of pixels in the foregroundand the background. Note that these variances can also be expressed as:

$\begin{matrix}{v_{F} = {{\sum\limits_{x\;\varepsilon\; F}{f(x)}^{2}} - {N_{F}\mu_{F}^{2}}}} & {{Eq}.\mspace{11mu} 21} \\{v_{B} = {{\sum\limits_{x\;\varepsilon\; B}{f(x)}^{2}} - {N_{B}\mu_{B}^{2}}}} & {{Eq}.\mspace{14mu} 22}\end{matrix}$A suitable partition F and B of S, which will minimize the sumE=ν_(F)+ν_(B) is to be found. Even with this simplification, the problemis still extraordinarily hard since there are still have 2^(N) possiblemasks. Thus, the image is further partitioned into 2×2 pixel sub-images.On each 2×2 sub-image, there are only 2⁴=16 possible masks, which meansthat on each of such regions, it is possible to determine optimal F andB, that minimize E=ν_(F)+ν_(B) (by trying all 16 combination and keepingthe one with smallest energy. It turns out that it is not necessary totry all 16 combinations. The problem is equivalent to a K-means problemwhere K=2, and since ƒ (the image) is a scalar function, the values ƒ(x)can be sorted which yield a solution, which can be computed veryefficiently: sort all 4 pixel value, and determine which one of the 3partitions (respecting the sorting) yield lowest energy. The partial sumcan be reused in each partition to minimize the number of operations.

This solution, however, has the drawback that every 2×2 sub-region has adistinct foreground and background which can pick up even the slightestpixel noise, resulting in an inadequate mask. The next step involvescombining adjacent regions. For example, supposing 2 regions, 1 and 2,and their corresponding foreground and background, F₁,B₁ and F₂,B₂. Whencombining these 4 sets, there are in effect seven distinctpossibilities:

TABLE 3 New F New B F1 B1, F2, B2 F1, B1, F2 B2 F1, F2, B2 B1 F2 F1, B1,B2 F1, F2 B1, B2 F1, B1 F1, F2 F1, B2 B1, F2The winning combination is the combination that has the lowest resultingenergy E=ν_(F)+ν_(B). Note that all seven combinations can be tried, orthe average in foregrounds and backgrounds can be sorted and only thepartitions that respect the sorting are considered. Furthermore, if eachregion retains quantities

${\sum\limits_{F}{f(x)}},{\sum\limits_{F}{f(x)}^{2}}$and N_(F) for the foreground and

${\sum\limits_{B}{f(x)}},{\sum\limits_{B}{f(x)}^{2}}$and N_(B), the combinations for E can be computed in constant time.After merging, these quantities must usually be recomputed, butfortunately, this is also accomplished in constant time. Also note thatthe sum Σƒ(x)² over the regions is constant for each partition, and neednot be calculated for the purpose of selecting an optimal partition.This sum Σƒ(x)² over the regions can also be employed to determine whennot to merge regions, as is shown below.

Combining adjacent regions can proceed by combining the horizontallyadjacent 2×2 regions into 2×4 regions, followed by combining thevertically adjacent 2×4 regions into 4×4 regions. The 4×4 regions arecombined into 8×8 regions and so on until only one region is left, whichis partitioned into foreground and background.

Unfortunately, such a technique can lead to merges which place severalgray levels into the foreground or into the background, with a potentialloss of important details such as text, whenever there are more than 2colors in a region. For example if region 1 has text written in grayover white, and region 2 is mostly black, the merge of the two regionscan lead to gray and white being placed in the foreground and blackbeing placed in the background of the resulting region, losing thetextual information. However, whenever two colors are merged in eitherforeground or background, a sharp increase of energy (or variance) isnoticed for that region, since a constant is no longer a good model forsuch region. Accordingly, it is desirable to employ an algorithm so asnot to merge adjacent region if the energy E exceed a certain thresholdK which is determined experimentally. This algorithm yields a mask,which captures most of the text and graphic lines in a page.

This algorithm however has a number of refinements—one of which is toemploy simple regions. A 1-region energy of small regions (for example4×4) can be measured and if the energy is sufficiently low (employinganother threshold determined experimentally), the entire region can beplaced into the foreground or background. Thus, partitioning of thesmall regions can, if the energy is lower than a threshold, be avoidedthereby accelerating the operation of the algorithm. Instead ofutilizing a constant assumption for foreground and background, apolynomial regression can be employed to represent the foreground andbackground. For example, if the polynomials are planes of equationαx+βy+μ, the energy would be defined by:

$\begin{matrix}{v_{F} = {\sum\limits_{x,{y\;\varepsilon\; F}}\left( {{f\left( {x,y} \right)} - {\alpha_{F}x} + {\beta_{\Gamma}y} + \mu_{F}} \right)^{2}}} & {{Eq}.\mspace{14mu} 23} \\{v_{B} = {\sum\limits_{x,{y\;\varepsilon\; B}}\left( {{f\left( {x,y} \right)} - {\alpha_{B}x} + {\beta_{B}y} + \mu_{B}} \right)^{2}}} & {{Eq}.\mspace{14mu} 24}\end{matrix}$Where x,y index the pixel locations, and α_(F),β_(F) and μ_(F) arescalars that minimize ν_(F) and α_(B),β_(B) and μ_(B) are scalars thatminimize ν_(B). Note that α_(F),β_(F) and μ_(F) can be solved inconstant time using the quantities Σƒ(x,y)²,Σƒ(x,y)x,Σƒ(x,y)y, andΣƒ(x,y). This is a linear system of three unknown and three equations,and the same applies to α_(B),β_(B) and μ_(B). As before, the algorithmis bottom-up and minimizes E at each merge. The foregrounds andbackgrounds cannot be sorted by average, and therefore all sevencombinations are tested to determine which combination minimizes E. Tokeep performing each test and merge in constant time, the quantitiesΣƒ(x,y)²,Σƒ(x,y)x,Σƒ(x,y)y,Σƒ(x,y) and N should be maintained for eachregion for the foreground and the background. The simple regionoptimization is still possible, but could assume a constant over theregion, a polynomial regression, or both.

A mask processor component 706 performs any number of processingoperations on the mask. The mask processor component includes a noiseremoval component 708, and a dither detector 710. It is appreciated thatalternate aspects of the invention can include other mask processingcomponents. It is further appreciated that alternate aspects of theinvention can include less or more processing components.

The noise removal component 708 removes noise from the mask. Asdiscussed above, digital documents are typically created by scanning instandard documents. A certain amount of noise is almost alwaysintroduced into the digital document via the scanning and digitizingprocess. For example, fingerprints on a scanner tray or a fold in adocument can cause noise and/or erroneous marks to be included in thedocument image. The noise removal component 708 first attempts toidentify noise in the mask. A variety of approaches can be employed toidentify noise in the mask. One approach is to analyze the mask forconnected components. Connected components are connected pixels of thesame color, for example black. The mask can be scanned left to right,top to bottom, page by page searching for connected components. Theconnected components can be stored with positional and shape or bitmapinformation. Then, each pixel of the document is analyzed to determinewhether it is noise or not. Thus, for each pixel, a surrounding region,such as all the pixels within a five pixels distance, is analyzed todetermine a number of connected components in and/or intersecting thesurrounding region. If the number of connected components is less than athreshold amount, the pixel is considered to be noise and removed fromthe mask. Then, the pixel is assigned to the foreground or backgroundimages of the document according to whichever provides bettercompression. Other approaches can be used and still be in accordancewith the present invention.

Generally, the noise removal component 708 can identify pixels thatappear to be random and/or unrelated to text or image in the document.It is appreciated that the noise removal component 708 does notnecessarily remove all the noise from the document. The noise removalcomponent 708 removes the identified noise from the mask by assigningthose pixels to the foreground image or the background image.Additionally, the noise can be removed by, for example, replacing pixelsidentified as noise with the values of surrounding pixels.

The dither detector 710 detects dithering in the mask and removesdithering from the mask. Dithering is a technique employed in computergraphics to create the illusion of varying shades of gray on amonochrome display or printer, or additional colors on a color displayor printer. Dithering relies on treating areas of an image as groups ofdots that are colored in different patterns. Akin to the print imagescalled halftones, dithering takes advantage of the eye's tendency toblur spots of different colors by averaging their effects and mergingthem into a single perceived shade or color. Depending on the ratio ofblack dots to white dots within a given area, the overall effect is of aparticular shade of gray. Dithering is employed to add realism tocomputer graphics and to soften jagged edges in curves and diagonallines at low resolutions. However, dithering can be problematic fordocument images. Dithering, if not appropriately detected and handled,can be misinterpreted as text, handwriting or a graphic.

The dither detector 710 identifies dithering regions in the mask—theregions can be divided such that each region or subregion has a similardithering pattern. The dither detector 710 removes or handles theidentified dithering regions. For example, the dither detector 710 couldidentify a dithering region as being a light gray. The dither detector710 could simply remove the dithering region, or set the region to alight gray. It is appreciated that changing a region to a shade wouldalso involve modifying the document image. The dither detector 710 canalso reallocate the identified dithering regions to the foreground orbackground by modifying the mask.

A number of approaches can be employed to detect and/or handledithering. One approach is to analyze the mask for connected components.Connected components are connected pixels of the same color, for exampleblack. The mask can be scanned left to right, top to bottom, page bypage searching for connected components. The connected components can bestored with positional and shape or bitmap information. For each pixelof the mask, a quantity or number of connected components in and/orintersecting a surrounding region is determined. The surrounding regioncan be a number of pixels surrounding the pixel, such as 7. If thenumber of connected components in the region is greater than athreshold, dithering is removed. Then, the pixel can be assigned toeither the foreground or background image, thus removing the dithering.This assignment can be determined by computing a quantity of pixels inthe background of the region and a quantity of pixels in the foregroundof the region. Then, the pixel can be assigned the foreground if thequantity of pixels in the foreground of the region is greater than thequantity of pixels in the background of the region, otherwise the pixelcan be assigned to the background. Other approaches for identifyingand/or handling dithering or half toning can be employed with thepresent invention.

A clustering component 712 identifies clusters in the mask. Theclustering component 712 locates connected components in the mask. Aconnected component, as described above is a plurality of connectedpixels, and can be either a four direction connected component or aneight direction connected component, for example. After locating theconnected components in the mask, the clustering component 712identifies similar connected components and groups them as clusters. Acluster is a group of similar connected components. Generally, a clusterrepresents an alphanumeric character, including handwriting, but caninclude other information as well. However, there can be a number ofclusters representing a single character. Clustering can dramaticallyincrease compression.

The reason for increase in compression and the reduction in file size,is that each connected component is summarized by a position, and apointer to a shape, belonging to a dictionary of shapes. The shape isthe bitmap or “image” of the connected component or cluster. Theconnected component is also referred to as a mark. The clustering aspectof the algorithm is to determine which shape should belong to thedictionary, and which shape is closest to each connected component.Typically the dictionary of shapes is a fraction of the file size, andcan even be shared across pages. The pointers to the shapes arecharacterized by a position in the page (X and Y), and a shape number.The X and Y position can be compressed using previous position, whilethe shape indices are compressed using context or a language model.

A number of approaches can be utilized by the clustering component 712to identify clusters. One approach is to compare shapes of marks to eachother and identify marks that are similar and/or identical. Anotherapproach employs properties of existing clusters and properties of marksor connected components. The existing clusters are groups of at leastone mark, that have previously been defined as clusters. Properties ofan extracted mark from a document are compared to the existing clusterproperties. Such comparison does not require a timely and expensive, bitby bit, comparison to identify mismatches. If a mark property fails tomatch any existing cluster properties, and if the mark is deemed to betoo far from any of the existing clusters, the mark is added as a newcluster and a bit by bit comparison is avoided. This approachdistinguish itself from other approaches by both a computationallyefficient screening for detecting mismatches, and a computationallyefficient algorithm for detecting positive matches for the clustering.

One particular property of the clusters that can be employed is x sizeand y size. The x size and y size provides sizing information of theexisting clusters. An x size and a y size of a mark can be compared tothe x size and y size of existing clusters to identify mismatches. Oneaspect of the invention is to organize the clusters in a 2D table ofbuckets, indexed by x size and y-size. When a new mark is found, thebucket of identical x size and y size is extracted, and the mark iscompared to the clusters in the bucket. Adjacent buckets can also besearched for more accurate matches (all the other buckets will beignored or screened). The “no drift” assumption, described infra, on thecluster center is useful here, because it guarantees that clusters stayin their bucket.

One property that can be employed for comparison of the mark to theexisting clusters is ink size. The ink size generally refers to a ratioof black pixels to total pixels in a mark or cluster. Similarly, an inksize of a mark can be compared to the ink size of existing clusters toidentify mismatches. Another property of the mark is a “hot point”. Thehot point is a location on the mark, which can be the center of gravity,or computed by other mean (for instance the line equation of thesurrounding characters. During comparison, the hot points of the markand the cluster are aligned before the comparison begins, oftenresulting in a translation. Yet another property is a reduced mark orimage, which is a pixel size reduced version a bitmap of the mark and/orcluster. In one aspect of the invention, the reduced mark is centered onthe hot point, and the mark is resealed so that the reduced mark has afixed size. Again, the non-drifting assumption of the cluster is usefulin guaranteeing that the reduced version of a cluster is a goodrepresentation of the all mark in the cluster. The reduced mark can becompared to the reduced mark or reduced image of existing clusters toidentify mismatches.

When a mark succeeds all the properties tests, it is then compared tothe cluster in a more direct way. A distance between the mark and thecluster is computed, and compared to a first threshold. If the distanceis within a threshold, the mark is added to the existing cluster. Themark can be added to the first acceptable existing cluster or the markcan be added to the existing cluster having the least distance from thecomparison with the mark.

Additionally, the clusters can be characterized by the first elementplaced in the cluster. An advantage of such approach is that adding anew mark to a cluster does not require re-computation of the cluster'scharacteristics. Another advantage of this approach is that it avoids“cluster drift”, which occurs when the cluster center moves with everyaddition of a new mark to the cluster. Not having cluster drift, alsoreferred to as the “no drift” assumption, yields guarantees on a maximumdistance between two elements of the same cluster—this guarantee affordsefficient and aggressive screening. The main drawback of such approachis that the clustering is not optimal in the sense that it will yieldmore clusters, for the same average cluster-to-mark distance, than otherclustering algorithm such as k-means. However, a significant advantageof this approach is increased speed. In the case of text compression,having a 10% increase in the number of clusters is not of concernbecause the size of the dictionary, which convey the information of eachclusters is typically a fraction of the size of the compressed document.

After the clustering component has 712 has identified clusters in themask, the mask is received by an OCR component 714 that performs opticalcharacter recognition (OCR) on the mask. The OCR component 714identifies possible characters in the mask, and can make use of clustersfrom the clustering component 707 to assist in analyzing the clustersfor characters. Each cluster is analyzed to determine if the cluster isa character, and then to determine what character the clusterrepresents. This approach can vary for different languages and/oralphabets. Generally, the shape or bitmap of the clusters are comparedto the shapes or bitmaps of the characters of the alphabet being used.If the shapes are similar enough, the cluster is recognized as thatparticular character. The position information of each character foundcan be maintained as well as their sequential order. The characters canbe stored as a sequence of characters of a standard alphabet, such asASCII.

A layout component 716 operates on the mask to improve compression andto provide layout information. The layout information can later beemployed to reflow the document image. The layout component 716 canutilize a number of approaches in order to improve compression and/orprovide layout information. One approach involves first organizingconnected components by color, vertical position and/or horizontalposition. The connected components can be provided by another componentof the system 700, such as the clustering component 712. The layoutcomponent 716 then joins or combines connected components based, atleast in part, upon color, vertical position, horizontal position, joindistance (e.g., distance between the connected components), height ofthe connected components and/or width of the connected components. Thus,joined connected components comprise two or more connected components.The joined components can also be joined with other connected componentsor other joined components. The layout component 716 then generates thelayout information to be provided. The layout information can includethe position of the joined components, locations of lines and otherinformation of the document image.

The mask encoder 718 receives the mask from the layout analysiscomponent 716 and encodes the mask to generate a compressed bitstreamreferred to as a mask bitstream. Any suitable compression scheme can beemployed by the mask encoder 718. The mask is binary and typicallyincludes textual information. Thus, a compression scheme should beselect that compresses binary and textual information well. A bi-levelcompression scheme can be employed to encode the mask. Additionally, themask encoder 718 encodes or supplies encoding information to be employedto decode the mask bitstream, such as, for example libraries,dictionaries, symbols, tables sizes and the like.

A compression scheme that can be employed to encode the mask is CCITT(Consultative Committee on International Telephone and Telegraph). TheCCITT is currently known as ITU-T International TelecommunicationsUnion-Telecommunications Sector (changed name in 1994)) which is astandards group and a name of a lossless compression technology forfax/modem communication. This type of compression works well with binaryimages. Typical compression ratios are 4:1 for the older versionV.42bis, and 6:1 for newer version V.44 2000, which is based on theLempel-Ziv-Jeff-Health (LZJH) compression algorithm. Other compressionmethods or schemes can be employed to encode the mask and still be inaccordance with the present invention.

A foreground background segmenter 719 receives the document image andthe mask and separates the document image into a foreground image and abackground image. According to this aspect of the invention, theforeground image and the background image are substantially the samesize as the document image (e.g., the document image being 640 by 480,the foreground image being 640 by 480 and the background image being 640by 480 pixels). However, alternate aspects of the invention can generateforeground and background images that differ substantially in size fromthe document image. The foreground image and background image are suchthat, when combined or recombined, can be identical or similar to thedocument image.

The foreground background segmenter 719 uses the mask, which is inbinary format, to create the foreground and background image. Forexample, all pixels represented by a 1 in the mask can go to theforeground image and all pixels represented by a 0 in the mask can go tothe background image. Conversely, as an example, all pixels representedby a 0 in the mask can go to the foreground image and all pixelsrepresented by a 1 in the mask can go to the background image. However,the background image and the foreground image typically have pixels withunassigned values. These pixels with unassigned values are referred toas holes, vacant regions and/or don't care regions.

Additionally, the foreground background segmenter 719 includes a dilatecomponent 720 and a retouch component 721. The dilate component 720operates on the mask to dilate the mask according to the foregroundimage and/or the background image. The dilate operation is amorphological operation based on the distance transform which takes amask and extend the regions of ‘1’ by k pixels, where k is a constant(e.g. 2). The distance can be a Manhattan distance or sphericaldistance. The converse operation, erosion, can be performed by adilation of the inverse mask. In typical scanned documents, text isoften washed out, as a result of ink bleeding, or low dpi setting duringthe scan. Therefore the transition from white to black is a continuousand gradual one, and the mask defines a hard boundary between foregroundand background. The continuous transition near the boundary on both theforeground and the background can introduce ringing and can be costly toencode. To alleviate this effect, it is advantageous to ignore the pixelvalues near the boundary. This is done by performing a dilation of the“don't care” values for both the foreground and the background, by adistance of several pixels (e.g. 2). Around the regions of text, thedilation of the “don't care” pixels increases both contrast andreadability, while reducing file size. When dilation is used in regionsof natural images, however, the effect of increased contrast makes theimages look cartoonish. To circumvent this problem, it is possible toperform a layout analysis of the mask and detect which regions are madeof text. The dilation can then be specific to text regions only. Thedilate component 720 can separate the mask into a foreground mask and abackground mask. As a result, there are pixels that are don't care forboth the foreground and the background. These pixels still get assigneda reasonable values because the dilate distance is small, and theforeground and background are typically smooth due to compression andtherefore the don't care pixel get assigned values that are not far offfrom neighboring do care pixels. However, care must be taken during thedilation of don't care not to completely erase small or thin region ofdo care pixels. If this happens, the pixels become unconstrained and faraway from any do care pixels, resulting in unpredictable coloring fromfar away regions. To prevent this from happening, one can look at thedistance transform and protect local extrema from being switch from docare to don't care.

It is appreciated, that the foreground image and the background imagecan eventually be recombined into a recombined document. However, thesharp edges and transitions can cause compression to degrade and cancause the recombined image to suffer visual deficiencies, such as edgesor lines. This can happen, for example, when the mask causes a spuriousboundary across a substantially similar color region, causing a portionof the substantially similar region to be placed in the foreground imageand another portion to be placed in the background image. Theseportions, when ultimately recombined, can have a visually, noticeableline across the spurious boundary because the portions are processedseparately and encoded separately. Additionally, the compression ofthese portions can be degraded because the spurious boundary is,essentially, encoded with each portion.

The retouch component 721 operates on the mask to “re-touch” thedocument image so that overall compression of the document image can beimproved and so that, visually, a recombined document image is improved.Generally, the retouch component 721 identifies spurious boundariesand/or extends the foreground and/or background image by way of themask. For one approach, the retouch component 721 utilizes the mask toidentify spurious boundaries in the foreground and background images.Detection can be performed by identifying horizontal and/or verticallines of a merged region boundary that are longer than a first thresholdquantity of pixels (e.g., five pixels) for which: (1) one side of theboundary is foreground and the other side background and (2) linearregression along that line on either side differs by more than a secondthreshold amount (e.g., linear regression can be performed on theboundary pixels of the foreground side of the pixels and linearregression performed for the pixels of the background side of thepixels). The retouch component 721 may obtain this region informationfrom the mask separator 704. The retouch component 721 then extends theforeground and/or background image by generating separate foreground andbackground masks from the mask. The foreground mask extends theforeground image while the background mask extends the background mask.Generally, the foreground image and the background image are extended anumber of pixels across spurious boundaries. It is appreciated that theforeground and background masks are not typically encoded or employed aspart of a decoding process. Other approaches can be employed for theretouch component 721 to improve compression and visual appearance ofthe document image, when recombined, and still be in accordance with thepresent invention.

As stated above, the foreground background segmenter 719 uses the maskto segment the document image into the foreground and background images.If foreground and background masks are generated by the dilate component720 and/or the retouch component, the foreground mask is employed tosegment the document image into the foreground image and the backgroundmask is employed to segment the document image into the backgroundimage.

A foreground pixel filler 722 receives the foreground image from theforeground background separator segmenter 719. Generally, the foregroundpixel filler 722 identifies don't care regions or vacant regions in theforeground image and fills these regions with don't care pixels. Theseregions are formed from pixels of the document image that are allocatedto the background image leaving vacant or don't care regions in theforeground image. The don't care pixels are not typically present in arecombined document image so the foreground pixel filler 722 does notconsider the visual appearance of the don't care pixels. Alternateaspects of the invention can avoid filling don't care pixels by using acompression scheme that properly handles the don't care regions, such asa masked wavelet compression scheme.

As stated above, the foreground image and the background image havevacant or don't care regions. The vacant regions can be handled a numberof ways. One approach is to fill the vacant regions with data and thenuse a regular compression technique. A simple process to fill the vacantregions of the images is to fill those vacant regions with an averagepixel value for that image. However, this process can create sharpdiscontinuities at the mask boundaries, can increase a required bit ratefor a given peak signal-to-noise ratio (PSNR) and produce noticeableringing near the mask or vacant region boundaries. Another process is tocolor each pixel with the color of the closest non-masked (or do careregion) pixel. A standard morphology algorithm allows that process to beperformed with only two passes over all the pixels leading toVoronoi-filled vacant regions under the mask. Then, when the foregroundor background image is reconstructed, the reconstructed image islow-passed and then the known pixels are restored to their correctvalues. If the low-pass filter cutoff frequency is too low, sharp edgescan occur causing an increase in the required bit rate and noticeableringing near boundaries.

Another approach to handling vacant regions is to use projection ontoconvex sets. For example, consider two convex sets: the set of imagesthat matches the input on the visible pixels, and the set of images thathave certain wavelet coefficients set to zero (e.g. all high-frequencycoefficients beyond a certain resolution level). By alternatingprojection onto those two sets, an image can be found that agrees withthe visible pixels and which compresses well because it has many zerowavelet coefficients.

Yet another approach to handling vacant regions is to use wavelettransforms designed explicitly for irregular grids. Such waveletdecompositions are needed in problems in computer vision and compressionof geometry data in computer graphics. Such wavelets are adapted to theirregular pattern of the mask.

Another approach is to use a masked wavelet compression scheme forencoding that can compress the foreground image well without filling thedon't care regions with don't care pixels. The masked waveletcompression scheme, as described with respect to FIG. 1, is one suchcompression scheme that can be employed. Other suitable approaches tohandling don't care regions can be used in accordance with the presentinvention.

A foreground downsample component 724 receives the foreground image fromthe foreground pixel filler 722 and downsamples the foreground image.The foreground downsample component 724 reduces the size of theforeground image to increase compression of the foreground image and thedocument image. For example, a foreground image of 640 by 480 pixels canbe downsampled to 320 by 240. A number of approaches can be employed todownsample the foreground image. One approach is to discard pixels todownsample the foreground image. Thus, for example, a foreground imagecan be downsampled from 640 by 480 to 320 by 240 pixels by discardingevery other pixel. Another approach is to generate new pixels based onan average or mean of corresponding pixels from the foreground image.For example, the new pixels of the foreground image can be respectivelygenerated as an average of eight neighboring pixels. Yet anotherapproach is to downsample using cubic interpolation. Still otherapproaches to downsample the image can be employed in accordance withthe present invention. If the foreground and background are downsampledin the encoder, they should generally be upsampled in the decoder beforebeing combined using the mask.

A foreground encoder 726 encodes the foreground image. The foregroundencoder 726 generates an encoded bitstream from the foreground image.This bitstream can be referred to as the foreground bitstream. Theforeground encoder 726 can utilize a number of suitable compressionschemes to encode the foreground image. Some exemplary compressionschemes that can be employed are, for example, progressive waveletencoding or progressive transform encoding.

The foreground encoder 726 can also include a constant color connectedcomponent analyzer (not shown) to further improve compression of theforeground image. The constant color connected component can useapproaches and algorithms similar to those described with respect to theclustering component 712. The constant color connected componentanalyzer scans the foreground image to identify constant color connectedpixels. If a group of constant color connected pixels is greater than athreshold value, that group is considered a constant color connectedcomponent or constant color mark. It is appreciated that “constantcolor” can include some variations in color. The constant colorconnected component analyzer can further group similar constant colorconnected components into constant color clusters. Related information,such as positional information for the constant color clusters, bitmaps,shapes, position information for the constant color connected componentsand the like can be included with the foreground bitstream. Constantcolor can also be restricted to text via a layout analyzer employed todetect which regions are composed of text.

A background pixel filler 730 receives the background image from thebackground background separator 719. The background pixel filler 730identifies don't care regions in the background image and fills theseregions with don't care pixels. These regions are formed from pixels ofthe document image that are allocated to the foreground image leavingvacant or don't care regions in the background image. The don't carepixels are not typically present in a recombined document image so thebackground pixel filler 730 does not consider the visual appearance ofthe don't care pixels. Alternate aspects of the invention can avoidfilling don't care pixels by using a compression scheme that properlyhandles the don't care regions, such as a masked wavelet compressionscheme. The background pixel filler 730 operates substantially the sameas the foreground pixel filler 722.

As stated above, the background image and the foreground image havevacant or don't care regions. The vacant regions can be handled a numberof ways. One approach is to fill the vacant regions with data and thenuse a regular compression technique. A simple process to fill the vacantregions of the images is to fill those vacant regions with an averagepixel value for that image. However, this process can create sharpdiscontinuities at the mask boundaries, can increase a required bit ratefor a given peak signal-to-noise ratio (PSNR) and produce noticeableringing near the mask or vacant region boundaries. Another process is tocolor each pixel with the color of the closest non-masked (or do careregion) pixel. A standard morphology algorithm allows that process to beperformed with only two passes over all the pixels leading toVoronoi-filled vacant regions under the mask. Then, when the foregroundor background image is reconstructed, the reconstructed image islow-passed and then the known pixels are restored to their correctvalues. If the low-pass filter cutoff frequency is too low, sharp edgescan occur causing an increase in the required bit rate and noticeableringing near boundaries.

Another approach to handling vacant regions is to use projection ontoconvex sets. For example, consider two convex sets: the set of imagesthat matches the input on the visible pixels, and the set of images thathave certain wavelet coefficients set to zero (e.g. all high-frequencycoefficients beyond a certain resolution level). By alternatingprojection onto those two sets, an image can be found that agrees withthe visible pixels and which compresses well because it has many zerowavelet coefficients.

Yet another approach to handling vacant regions is to use wavelettransforms designed explicitly for irregular grids. Such waveletdecompositions are needed in problems in computer vision and compressionof geometry data in computer graphics. Such wavelets are adapted to theirregular pattern of the mask.

Another approach is to use a masked wavelet compression scheme forencoding that can compress the background image well without filling thedon't care regions with don't care pixels. The masked waveletcompression scheme, as described with respect to FIG. 1, is one suchcompression scheme that can be employed. Other suitable approaches tohandling don't care regions can be employed in accordance with thepresent invention.

A background downsample component 732 receives the background image fromthe background pixel filler 730 and downsamples the background image.The background downsample component 732 reduces the size of thebackground image to increase compression of the background image and thedocument image. For example, a background image of 640 by 480 pixels canbe downsampled to 320 by 240. A number of approaches can be employed todownsample the background image. One approach is to discard pixels todownsample the background image. Thus, for example, a background imagecan be downsampled from 640 by 480 to 320 by 240 pixels by discardingevery other pixel. Another approach is to generate new pixels based onan average or mean of corresponding pixels from the background image.For example, the new pixels of the background image can be respectivelygenerated as an average of eight neighboring pixels. Still otherapproaches to downsample the image can be employed in accordance withthe present invention. Generally, the size of the foreground image andthe background images are substantially the same after being downsizedby the foreground downsample component 724 and the background downsamplecomponent 732, respectively.

A background encoder 734 encodes the background image. The backgroundencoder 734 generates an encoded bitstream from the background image.This bitstream can be referred to as the background bitstream. Thebackground encoder 734 can utilize a number of suitable compressionschemes to encode the background image. Some exemplary compressionschemes that can be employed are, for example, progressive waveletencoding or progressive transform encoding.

The background encoder 734 can also include a constant color connectedcomponent analyzer (not shown) to further improve compression of thebackground image. The constant color connected component can useapproaches and algorithms similar to those described with respect to theclustering component 712. The constant color connected componentanalyzer scans the background image to identify constant color connectedpixels. If a group of constant color connected pixels is greater than athreshold value, that group is considered a constant color connectedcomponent or constant color mark. It is appreciated that “constantcolor” can include some variations in color. The constant colorconnected component analyzer can further group similar constant colorconnected components into constant color clusters. Related information,such as positional information for the constant color clusters, bitmaps,shapes, position information for the constant color connected componentsand the like can be included with the background bitstream. Constantcolor can also be restricted to text, where a layout analyzer is used todetect which regions are composed of text.

Any suitable number of compression schemes can be employed with the maskencoder 718, the foreground encoder 726 and the background encoder 734.Some of these schemes have been described above. Additional details ofthose compression schemes and additional compression schemes that can beemployed with the mask encoder 718, the foreground encoder 726 and thebackground encoder 734 are described here.

Some commonly known compression schemes that can be employed with thepresent invention are JPEG (joint photographic experts group, PWT(progressive wavelet transform), JPEG 2000, PTC (progressive transformcodec), GIF (graphic interchange format), PNG (portable networkgraphics) can be employed. JPEG is, in general, a lossy compressionbased on Discrete Cosine Transformation (DCT), although it does havesettings for lossless compression, which achieves a typical compressionratio of 2:1. This compression technology is generally used forphotographic images. JPEG can store 24-bit per-pixel color (16 millioncolors) at an average compression ratio of 20:1 for full colorcompression. Generally, however, average compression ratio varies from10:1 to 20:1 without visible loss, 60:1–100:1 for moderate quality, and60:1 to 100:1 for poor quality. A few drawbacks of this technology areits difficulty with sharp edges, and its lack of support fortransparency. Additionally, it has takes longer to view than GIF format,but it is superior to GIF with regard to images like full color orgray-scale scanned photos.

PWT is part of lossless technology and is the basis for many othercompression technologies. PWT is based on the idea that it is best tosee the entire image first and continue to increase the resolutioniteratively over time. This allows for efficient transfer of highfidelity images. Specifically, by transferring the low frequencycomponents first, a low-resolution version of the entire image istransferred. Then the resolution is increased as the higher frequencycomponents arrive.

JPEG 2000 was designed to replace JPEG standard based on DCT withwavelet based compression technology. JPEG 2000 eliminates blockingartifacts present in JPEG, while maintaining 24-bit color. Additionally,JPEG 2000 supports both lossy and lossless compression with compressionratios of 201:1 for lossy and 2:1 for lossless.

PTC is a compression scheme that combines the some of the features offormats such as JPEG, GIF, PNG, M-JPEG, JPEG-LS, and others, but withbetter compression performance and in many cases significantly fasterencoding and decoding. PTC can provide a reduced amount of artifactingcompared to JPEG.

PTC also adds new features. One important added feature is scalability(or progressive rendering). From an original PTC file, alower-resolution or lower-fidelity picture can be obtained by parsingand decoding a portion of the file. PTC also supports up to 16 bits percolor and four color channels or components (R, G, B, and alpha), wherealpha is a transparency component. PTC also integrates lossless andlossy encoding in a single codec. Additionally, efficient reference codefor PTC is available. The code has low memory footprint and uses onlyinteger arithmetic (no floating-point), for exact reproducibility.

In order to implement PTC, a multi-resolution transform is employed toachieve progressive rendering. One approach is to use wavelets, but amore efficient one, with a lower computation cost, is to usehierarchical lapped transforms. In PTC a new hierarchical lappedbiorthogonal transform (LBT), which improves on prior PTC constructionin by using a smaller 4×4 block size, and integer coefficients—allowingfor its computation without substantially any multiplications (thetransform in prior PTC schemes still employed floating-pointarithmetic). The main advantage of lapped transforms over the populardiscrete cosine transform (DCT, used in JPEG and MPEG) is that lappedtransforms are free from the “blocking artifacts” of the DCT. Also, dueto the multiresolution (hierarchical) form of the LBT used in PTC,ringing artifacts are less noticeable than in DCT-based codecs.

GIF is a lossless 8-bits-per-pixel color (256 colors) compression formatwith an average compression of about 3:1. GIF has Lempel-Ziv-Welch (LZW)coding built in, which is why it does not provide great compression forphotos but it does for simpler line drawings. GIF compression can workwith all kinds of images, but it works best with computer-generatedgraphics with only a few colors, which makes it is popular for Web basedgraphics. Moreover, diagrammatical images with sharp edges arecompressed better than JPEG.

PNG is a lossless format that offers compression on average 10–30%smaller than GIF. It is also better than JPEG for true color images,because it has 48-bit color, which is better than JPEG's 24-bit color.Additionally, PNG allows for transparency, and its compression isasymmetric, meaning its decompression is fast while compression takesmuch longer. PNG is designed for use on the Internet so its fastdecompression is plus. Additionally, PNG was designed to replace the GIFformat, which uses LZW coding. PNG uses the freeware Gzip compressionscheme instead of LZW coding.

FIG. 8 is a block diagram of a segmented layered image decoding systemaccording to one aspect of the invention. The system receives compressedbits, bitstream or file and generates a recombined image. The systemincludes a separator component 802, a foreground decoder 804, abackground decoder 806, a mask decoder 808, a re-flow component 810 anda combiner 812.

The separator component 802 receives the compressed bits and separatesthe bits to form a foreground bitstream, background bitstream and maskbitstream. The separator component 802 can utilize header information toseparate the bitstreams. The foreground decoder 804 decompresses theforeground bitstream to yield a foreground image. The background decoder806 decompresses the background bitstream to yield a background image.

The mask decoder 808 decompresses the mask bitstream to yield a mask ormask image. The mask decoder 808 can also retrieve layout and/or re-flowinformation. Additionally, the mask decoder 808 can obtain a dictionaryemployed for reconstructing textual information. For some compressionschemes (e.g., masked wavelet), the foreground decoder 804 and thebackground decoder 806 can require the decompressed mask from the maskdecoder 808 in order to obtain the foreground image and the backgroundimage.

The re-flow component 810 operates on the foreground image, thebackground image and the mask according to desired display properties.The desired display properties can include properties such as displaypage size, number of columns, font size and the like. The desireddisplay properties can be quite different than the properties of theoriginal document image. For example, a document image can have a letterpage size and a 10 point font size. Those properties of the documentimage are fine for viewing on standard letter paper. However, thoseproperties would likely make viewing on a portable device such as aportable digital assistant (PDA) difficult at best. The PDA, generallyhas a narrow screen and lower resolution. Without re-flowing, a userusing the PDA would have to pan across to read each line of text. Thus,for the PDA, the desired display properties may be a column size oftwenty, font size of 14 and the like. The re-flow component 810 can wraptext as necessary, reposition paragraphs, reposition sentences,reposition images, resize images and/or perform any other suitablemodifications according to the desired display properties. The re-flowcomponent 810 utilizes layout information encoded with the foregroundimage, background image and/or mask.

The combiner 812 combines the foreground image, the background image andthe mask into a recombined document image. The recombined image isidentical to or an approximation to an original document image.Additionally, the recombined image can have a different shape and/orsize depending on the desired display properties. The combiner 812generates pixels for the recombined document image from the foregroundand background images based on the mask. Each pixel of the recombineddocument image is determined by referencing a corresponding pixel of themask thereby determining whether the pixel of the recombined documentshould come from the foreground image or the background image. Forexample, a mask pixel value of 1 can indicate that the pixel of therecombined document should come from the corresponding pixel of theforeground image. Additionally, the combiner 812 performs any suitablecolor space conversion necessary. For example, the recombined image maybe represented using the YCoCg colorspace. Thus, continuing the example,the combiner 812 would convert from the YCoCg colorspace to the RGBcolorspace (e.g. in order to display on a monitor).

FIG. 9 is a flow diagram of a method of encoding a document imageaccording to one aspect of the invention. A document image is providedat 902. The document image is a bitmap image and has a colorspacerepresentation, which typically is RGB. The document image can beconverted from its colorspace to another colorspace, such as YUV orYCoCg, to increase compression.

A mask is generated for the document image at 904. The mask is a binaryimage based on the document image. The mask is generated such that thedocument image can be segmented or separated into a foreground image anda background image such that the foreground image and the backgroundimage are highly compressible. Each pixel of the mask determines whethera corresponding pixel of the document image is placed in the foregroundimage or the background image. The mask can be generated in any suitablemanner such that a combined size of the mask, the foreground image andthe background image is reduced after compression.

It is appreciated that additional processing of the mask, including butnot limited to, noise detection, noise correction, dither detection andhandling and the like can be performed on the mask. For example, noiseis often introduced into a document image during a digitization process.Fingerprints and/or dirt, for example, can be included unintentionallyin the document image. The noise can be identified and removed from themask. Dithering is a technique employed to create the illusion ofvarying shades of gray or colors. Dithering varies a ratio or percentageof dots in a region to simulate the shade. Dithering may make an imageappear nicely, but can cause problems for digital documents. Thedithering can get misinterpreted as text, handwriting and the like. Thedithering in the mask can be identified and appropriately handled.

Additionally, clusters can be identified in the mask. Connectedcomponents are identified in the mask. Connected components are groupsof one or more connected dots, typically black dots. Similar connectedcomponents are grouped into clusters. Each cluster can be representedwith a bitmap and pointers to locations in the mask. The use of clusterscan save a tremendous amount of space. Clusters can representcharacters, handwriting and the like.

Also, optical character recognition can be performed on the mask. Theclusters, whether representing handwriting or text, can be analyzed toidentify text or characters. Once recognized, the characters can bestored as, for example, ASCII code and save even more storage space.Furthermore, optical character recognition enhances re-flow and ease ofediting of the document image when the document image is ultimatelyrecombined.

The mask can also be analyzed to identify layout properties of thedocument image. Characters, clusters and images positions can beanalyzed to identify the layout properties. Layout properties includingbut not limited to, sentences, paragraphs, forms, image positions,pagination and the like can be identified. The layout properties can beattached or encoded with the mask for later use.

The document image is segmented into a foreground image and a backgroundat 906, according to the mask. For example, all pixels represented by a1 in the mask can go to the foreground image and all pixels representedby a 0 can go to the background image. It is appreciated that otheraspects of the invention can segment the document image into anothersuitable number of layers and that the invention is not limited to thetwo layers described with respect to FIG. 9.

The foreground image and the background image are the same size as thedocument image. However, the foreground image and the background imagehave pixels that have not been assigned a value. Those pixels arereferred to as holes, voids and/or vacant regions. The holes occur wherethe values corresponding to those pixels were assigned to the otherimage. Retouching can be performed on the background and foregroundimages to reduce hard edges in the ultimately recombined image.Retouching involves extending the foreground image and the backgroundimage by a number of pixels so that sharp edges and/or transitions areavoided. Typically, the retouching is performed by generating aforeground mask and a background mask from the mask and segmenting theforeground image from the document image using the foreground mask andsegmenting the background image from the document image using thebackground mask. The foreground mask facilitates extending of theforeground image and the background mask facilitates extending of thebackground image.

Holes or voids in the foreground image are filled with don't care pixelsat 908. The don't care pixels are selected so as to increase compressionof the foreground image and reduce the size of the foreground image,after compression. The don't care pixels can be determined using anumber of suitable approaches. One approach that can be employed is tofill don't care regions with an average pixel value for the documentimage. However, this process can create sharp discontinuities at themask boundaries, can increase a required bit rate for a given peaksignal-to-noise ratio (PSNR) and produce noticeable ringing near themask or vacant region boundaries. Another approach is to color eachpixel in the don't care regions with the color of the closest non-masked(or do care region) pixel. A standard morphology algorithm allows thatprocess to be performed with only two passes over all the pixels leadingto Voronoi-filled vacant regions under the mask. Then, when theforeground or background image is reconstructed, the reconstructed imageis low-passed and then the known pixels are restored to their correctvalues. If the low-pass filter cutoff frequency is too low, sharp edgescan occur causing an increase in the required bit rate and noticeableringing near boundaries. Another approach to handling vacant regions isto use projection onto convex sets. For example, consider two convexsets: the set of images that matches the input on the visible pixels,and the set of images that have certain wavelet coefficients set to zero(e.g. all high-frequency coefficients beyond a certain resolutionlevel). By alternating projection onto those two sets, an image can befound that agrees with the visible pixels and which compresses wellbecause it has many zero wavelet coefficients. Other suitable approachesof filling the don't care regions can be employed in accordance with thepresent invention.

Holes or voids in the background image are filled with don't care pixelsat 910. The don't care pixels are selected so as to increase compressionof the background image and reduce the size of the background image,after compression. The don't care pixels can be determined using asuitable approach as described above with respect to the foregroundimage.

It is appreciated that alternate aspects of the invention can leavevacant regions unfilled. Thus, a compression scheme, such as the maskedwavelet compression scheme, can be employed to encode the foreground andbackground images without filling the vacant or don't care regions.

The mask is encoded to generate a mask bitstream at 912. The mask isencoded using a suitable compression scheme. The compression scheme canutilize the textual information and binary representation of the mask inorder to improve compression of the mask. Additionally, clusters, layoutinformation and the like can also be employed to better compress themask.

It is appreciated that additional processing can be performed on thebackground and foreground images. For example, noise removal anddownsampling can be performed on the background and foreground images.Noise removal removes or handles noise present in the foreground andbackground images. Typically, this noise is introduced in a scanning orconversion process. Downsampling reduces the size of the foreground andbackground images. For example, 640 by 480 pixel foreground andbackground images can be downsampled to 320 by 240 sized images.

The foreground image is encoded to generate a foreground bitstream at914. The foreground image is encoded using a suitable compressionscheme. Some exemplary compression schemes that can be employed toencode the foreground image are PWC, PTC, JPEG, JPEG 200, PNG and GIF.Other compression schemes can be employed and still be in accordancewith the present invention.

The background image is encoded to generate a background bitstream at916. The background image is encoded using a suitable compressionscheme. Some exemplary compression schemes that can be employed toencode the foreground image are PWC, PTC, JPEG, JPEG 200, PNG and GIF.Other compression schemes can be employed and still be in accordancewith the present invention.

The background and foreground images can also be analyzed to identifyconstant color connected components in order to further improvecompression of the foreground and background images. Constant colorconnected components comprise a plurality of connected pixels havingsubstantially the same color value. The constant color connectedcomponents can be identified by locating a seed pixel of a color andusing an algorithm to find all, or substantially all of the connectedpixels having that same color or substantially same color.

The mask bitstream, the foreground bitstream and the backgroundbitstream are combined at 918 to form a combined bitstream. The combinedbitstream can also be a file. Header information can be added to thecombined bitstream to include information such as encoder used,dictionaries, symbols, libraries and the like.

FIG. 10 is a flow diagram of a method of decoding a document imageaccording to one aspect of the invention. A bitstream is received at1002. The bitstream is a combined bitstream including a mask bitstream,a foreground bitstream, a background bitstream and header information.The bitstream is separated into the mask bitstream, the foregroundbitstream and the background bitstream at 1004. Additional, the headerinformation can be extracted.

A mask, a foreground image and a background image are decoded from themask bitstream, the foreground bitstream and the background bitstream,respectively at 1006. The mask, the foreground image and the backgroundimage are collectively referred to as the image components. The mask,the foreground image and the background image are decoded using adecompression scheme compatible with the compression scheme of whichthey were encoded with. The header information can include thiscompression/decompression information. The mask and/or headerinformation can include layout information or layout properties.Additionally, dictionaries, symbols and other information can be decodedfrom the mask or obtained from the header information.

Display properties are provided at 1008. The display properties can bepredetermined, input by a user and/or dynamically determined. Thedisplay properties include information such as display size, font size,columns and the like. The mask, the foreground image and the backgroundimage are modified according to the display properties at 1010. Theimage components are modified so that when recombined, they match thedisplay properties. For example, text in the mask can be re-flowed toproperly display on a narrow screen without panning across.

The mask, the foreground image and the background image are combined at1012 to form a recombined document image. The foreground image and thebackground image are combined according to the mask to form therecombined document image. The mask is binary, thus each pixel of therecombined document image is assigned a value from a corresponding pixelof the foreground image or the background image depending on acorresponding pixel of the mask. The recombined document image isidentical to or a close approximation of an original document image. Theoriginal document image is the document image originally encoded intothe received bitstream.

It is to be appreciated that the systems and/or methods of the presentinvention can be utilized in an overall system facilitatingidentification and/or compression of text, handwriting, drawings and thelike. Further, those skilled in the art will recognize that the systemand/or method of the present invention can be employed in a vast arrayof document image applications, including, but not limited to,photocopiers, document scanners, optical character recognition systems,PDAs, fax machines, digital cameras and/or digital video cameras.

In order to provide additional context for various aspects of thepresent invention, FIG. 11 and the following discussion are intended toprovide a brief, general description of one possible suitable computingenvironment 1110 in which the various aspects of the present inventionmay be implemented. It is to be appreciated that the computingenvironment 1110 is but one possible computing environment and is notintended to limit the computing environments with which the presentinvention can be employed. While the invention has been described abovein the general context of computer-executable instructions that may runon one or more computers, it is to be recognized that the invention alsomay be implemented in combination with other program modules and/or as acombination of hardware and software. Generally, program modules includeroutines, programs, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Moreover,one will appreciate that the inventive methods may be practiced withother computer system configurations, including single-processor ormultiprocessor computer systems, minicomputers, mainframe computers, aswell as personal computers, hand-held computing devices,microprocessor-based or programmable consumer electronics, and the like,each of which may be operatively coupled to one or more associateddevices. The illustrated aspects of the invention may also be practicedin distributed computing environments where certain tasks are performedby remote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in both local and remote memory storage devices.

FIG. 11 illustrates one possible hardware configuration to support thesystems and methods described herein. It is to be appreciated thatalthough a standalone architecture is illustrated, that any suitablecomputing environment can be employed in accordance with the presentinvention. For example, computing architectures including, but notlimited to, stand alone, multiprocessor, distributed, client/server,minicomputer, mainframe, supercomputer, digital and analog can beemployed in accordance with the present invention.

With reference to FIG. 11, an exemplary environment 1110 forimplementing various aspects of the invention includes a computer 1112,including a processing unit 1114, a system memory 1116, and a system bus1118 that couples various system components including the system memoryto the processing unit 1114. The processing unit 1114 may be any ofvarious commercially available processors. Dual microprocessors andother multi-processor architectures also can be used as the processingunit 1114.

The system bus 1118 may be any of several types of bus structureincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of commercially available busarchitectures. The computer memory 1116 includes read only memory (ROM)1120 and random access memory (RAM) 1122 A basic input/output system(BIOS), containing the basic routines that help to transfer informationbetween elements within the computer 1112, such as during start-up, isstored in ROM 1120.

The computer 1112 may further include a hard disk drive 1124, a magneticdisk drive 1126, e.g., to read from or write to a removable disk 1128,and an optical disk drive 830, e.g., for reading a CD-ROM disk 1132 orto read from or write to other optical media. The hard disk drive 1124,magnetic disk drive 1126, and optical disk drive 1130 are connected tothe system bus 1118 by a hard disk drive interface 1134, a magnetic diskdrive interface 1136, and an optical drive interface 1138, respectively.The computer 1112 typically includes at least some form of computerreadable media. Computer readable media can be any available media thatcan be accessed by the computer 1112. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other magnetic storage devices, or any other medium which can be usedto store the desired information and which can be accessed by thecomputer 1112. Communication media typically embodies computer readableinstructions, data structures, program modules or other data in amodulated data signal such as a carrier wave or other transportmechanism and includes any information delivery media. The term“modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and otherwireless media. Combinations of any of the above should also be includedwithin the scope of computer readable media.

A number of program modules may be stored in the drives and RAM 1122,including an operating system 1140, one or more application programs1142, other program modules 1144, and program non-interrupt data 1146.The operating system 1140 in the computer 1112 can be any of a number ofcommercially available operating systems.

A user may enter commands and information into the computer 1112 througha keyboard 1148 and a pointing device, such as a mouse 1150. Other inputdevices (not shown) may include a microphone, an IR remote control, ajoystick, a game pad, a satellite dish, a scanner, or the like. Theseand other input devices are often connected to the processing unit 1114through a serial port interface 1152 that is coupled to the system bus1118, but may be connected by other interfaces, such as a parallel port,a game port, a universal serial bus (“USB”), an IR interface, etc. Amonitor 1154, or other type of display device, is also connected to thesystem bus 1118 via an interface, such as a video adapter 1156. Inaddition to the monitor, a computer typically includes other peripheraloutput devices (not shown), such as speakers, printers etc.

The computer 1112 may operate in a networked environment using logicaland/or physical connections to one or more remote computers, such as aremote computer(s) 1158. The remote computer(s) 1158 may be aworkstation, a server computer, a router, a personal computer,microprocessor based entertainment appliance, a peer device or othercommon network node, and typically includes many or all of the elementsdescribed relative to the computer 1112, although, for purposes ofbrevity, only a memory storage device 1160 is illustrated. The logicalconnections depicted include a local area network (LAN) 1162 and a widearea network (WAN) 1164. Such networking environments are commonplace inoffices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 1112 isconnected to the local network 1162 through a network interface oradapter 1166. When used in a WAN networking environment, the computer1112 typically includes a modem 1168, or is connected to acommunications server on the LAN, or has other means for establishingcommunications over the WAN 1164, such as the Internet. The modem 1168,which may be internal or external, is connected to the system bus 1118via the serial port interface 1152. In a networked environment, programmodules depicted relative to the computer 1112, or portions thereof, maybe stored in the remote memory storage device 1160. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

FIG. 12 is a schematic block diagram of a sample computing environment1200 with which the present invention can interact. The system 1200includes one or more client(s) 1210. The client(s) 1210 can be hardwareand/or software (e.g., threads, processes, computing devices). Thesystem 1200 also includes one or more server(s) 1230. The server(s) 1230can also be hardware and/or software (e.g., threads, processes,computing devices). The servers 1230 can house threads to performtransformations by employing the present invention, for example. Onepossible communication between a client 1210 and a server 1230 may be inthe form of a data packet adapted to be transmitted between two or morecomputer processes. The system 1200 includes a communication framework1250 that can be employed to facilitate communications between theclient(s) 1210 and the server(s) 1230. The client(s) 1210 are operablyconnected to one or more client data store(s) 1260 that can be employedto store information local to the client(s) 1210. Similarly, theserver(s) 1230 are operably connected to one or more server datastore(s) 1240 that can be employed to store information local to theservers 1230.

What has been described above includes examples of the presentinvention. It is, of course, not possible to describe every conceivablecombination of components or methodologies for purposes of describingthe present invention, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of the presentinvention are possible. Accordingly, the present invention is intendedto embrace all such alterations, modifications and variations that fallwithin the spirit and scope of the appended claims. Furthermore, to theextent that the term “includes” is used in either the detaileddescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprising” as “comprising” is interpretedwhen employed as a transitional word in a claim.

1. A document encoding system comprising: a mask separator that receivesa document image and generates a mask from the document image, the maskbeing generated at least in part via merging regions by pairs of thedocument image based at least in pan upon minimization of pixel energy,the mask generated so as to reduce an estimated compression of aforeground image and a background image, the estimated compressioncomprising an energy variance computation; and an image segmenter thatsegments the document image into a plurality of layers according to themask, the plurality of layers represented in non-binary format.
 2. Thedocument encoding system of claim 1, the plurality of layers comprisinga first image layer and a second image layer and the mask represented inbinary format.
 3. The document encoding system of claim 2, the firstimage layer being a foreground image and the second image layer being abackground image.
 4. The document encoding system of claim 3, ftrthercomprising a mask encoder to encode the mask into a mask bitstream. 5.The document encoding system of claim 4, the mask encoder utilizing afax based compression scheme.
 6. The document encoding system of claim4, the mask encoder utilizing a bi-level compression scheme.
 7. Thedocument encoding system of claim 4, further comprising: a foregroundencoder that encodes the foreground image from the image segmcnter intoa foreground bitstream; and a background encoder that encodes thebackground image from the image segmenter into a background bitstrearn.8. The document encoding system of claim 7, the foreground encoder andthe background encoder utilizing a progressive wavelet transformcompression scheme.
 9. The document encoding system of claim 7, theforeground encoder and the background encoder utilizing a progressivetransform compression scheme.
 10. The document encoding system of claim7, the foreground encoder and the background encoder utilizing a maskedwavelet compression scheme.
 11. The document encoding system of claim 7,further comprising a combiner component to combine the mask bitstream,the foreground bitstream and the background bitstream into a combinedbitstream.
 12. A document encoding system comprising: a mask separatorthat receives a document image and generates a mask from the documentimage, the mask generated so as to reduce an estimated compression of aforeground image and a background images, the estimated compressioncomprising an energy variance computation; a retouch component thatidentifies spurious boundaries and/or extends the foreground and/orbackground image by way of the mask; and an image segmenter thatsegments the document image into a plurality of layers according to themask, the plurality of layers represented in non-binary format.
 13. Thedocument encoding system of claim 12, the retouch component beingoperative to generate a foreground mask and a background mask based onthe identified spurious boundaries and the image segmenter utilizing theforeground mask to obtain a foreground image and the background mask toobtain a background image.
 14. The document encoding system of claim 12,further comprising: a mask encoder to encode the mask into a maskbitstream; a foreground encoder that encodes the foreground image fromthe image segmenter into a foreground bitstream; and a backgroundencoder that encodes the background image from the image segmenter intoa background bitstream.
 15. A document encoding system comprising: amask separator that receives a document image and generates a mask fromthe document image, the mask represented in binary format and generatedat least in part via merging regions by pairs of the document imagebased at least in part upon minimization of pixel energy; and aforeground background segmenter that segments the document image into aforeground image and a background image according to the mask, theforeground image and the background image represented in a non-binaryformat, the mask generated so as to reduce an estimated compression ofthe foreground image and the background image, the estimated compressioncomprising an energy variance computation.
 16. The system of claim 15,the mask generated to reduce a combined size of the mask, the foregroundimage and the background image.
 17. The system of claim 15, theforeground image comprising a plurality of colors and the backgroundimage comprising a plurality of colors.
 18. The system of claim 15, thedocument image comprising textual information.
 19. The system of claim15, the document image comprising a single page.
 20. The system of claim15, the document image comprising handwriting.
 21. A photocopieremploying the system of claim
 15. 22. A document scanner employing thesystem of claim
 15. 23. An optical character recognition systememploying the system of claim
 15. 24. A personal digital assistantemploying the system of claim
 15. 25. A fax machine employing the systemof claim
 15. 26. A digital camera employing the system of claim
 15. 27.A digital video camera employing the system of claim
 15. 28. A segmentedlayered image system employing the system of claim
 15. 29. A video gameemploying the system of claim
 15. 30. A document encoding systemcomprising: a colorspace converter that converts a document image havinga first colorspace to a converted document image having a secondcolorspace; a mask separator that generate a mask from the converteddocument image, the mask represented in binary format and beinggenerated at least in part via merging regions by pairs of the documentimage based at least in part upon minimization of pixel energy; and aforeground background segnienter that segments the convened documentimage into a foreground image and a background image according to themask, the foreground image and the background image represented in anon-binary format, the mask generated so as to reduce an estimatedcompression of the foreground image and the background image, theestimated compression comprising an energy variance computation.
 31. Thesystem of claim 30, the first colorspace being RGB and the secondcolorspace being YUV.
 32. The system of claim 30, the first colorspacebeing RGB and the second colorspace being YCoCg.
 33. A document decodingsystem comprising: a separator component that separates a bitstream intoa foreground bitstream, a background bitstream and a mask bitstream; aforeground decoder that decodes the foreground bitstream into aforeground image; a background decoder that decodes the backgroundbitstream into a background image; a mask decoder that decodes the maskbitstream into a mask, the mask generated so as to reduce an estimatedcompression of the foreground image and the background image, theestimated compression comprising an energy variance computation; and acombiner component that generates a document image as a combination ofthe foreground image and the background image according to the mask. 34.The system of claim 33, the document image having a pluarlity of pixels,each of the plurality of pixels obtained from one of the groupcomprising the foreground image and the background image.
 35. The systemof claim 33, the document image being identical to an original documentimage.
 36. The system of claim 33, the document image beingsubstantially similar to an original document image.
 37. The system ofclaim 36, further comprising a segmented layer document encoding systemto generate the mask, the foreground image and the background image, toencode the mask into the mask bitstream, to encode the foreground imageinto the foreground bitstream, to encode the background image into thebackground image and to combine the mask bitstream, the foregroundbitstream and the background bitstream into the bitstream.
 38. Adocument system comprising: a colorspace converter that converts acolorspace of a document image; a mask separator that receives thedocument image from the colorspace converter and generates a mask forthe document image at least in part via merging regions by pairs of thedocument image based at least in part upon minimization of pixel energy;a mask processor component that receives the mask from the maskseparator and to process the mask; and a segmenter that segments thedocument image into a foreground image and a background image accordingto the mask, the mask generated so as to reduce an estimated compressionof the foreground image and the background image, the estimatedcompression comprising an energy variance computation.
 39. The system ofclaim 38, the mask processor component comprising a dither detectioncomponent to identify and remove dithering from the mask.
 40. The systemof claim 38, the mask processor component further comprising a noiseremoval component to identify and remove noise from the mask.
 41. Thesystem of claim 38, further comprising a clustering component thatreceives the mask from the mask processor component and determinesconnected components in the mask and groups similar connected componentstogether as clusters utilizing at least one screening property.
 42. Thesystem of claim 41, further comprising a layout component that receivesthe mask from the clustering component and identifies layout informationfrom the mask.
 43. The system of claim 42, further comprising a maskencoder that receives the mask from the layout component and encodes themask into a mask bitstream.
 44. The system of daixn 43, the segmentercomprising a retouch component that analyzes the mask and the documentimage for spurious boundaries and extends a foreground mask and abackground mask by a number of pixels across the spurious boundaries,the foreground mask and the background mask based on the mask, theforeground mask utilized by the segmenter to obtain the foreground imageand the background mask utilized by the segmenter to obtain thebackground image.
 45. The system of claim 44, further comprising a colorconnected component that identifies color connected components in theforeground image and the background image.
 46. The system of claim 44,further comprising a foreground encoder that receives the foregroundimage and encodes the foreground image into a foreground bitstream; abackground encoder that receives the background image and encodes thebackground image into a background bitstream; and a combiner componenttat combines the mask bitstream, the foreground bitstream and thebackground bitstream into a combined bitstream.
 47. The system of claim46, further comprising an optical character recognition component thatreceives the mask from the clustering component and identifiescharacters in the mask.
 48. The system of claim 47, further comprising aforeground pixel filler and a background pixel filler, the foregroundpixel filler fills don't care regions of the foreground image with don'tcare pixels and the background pixel filler fills don't care regions ofthe background image with don't care pixels.
 49. A method of encoding adocument comprising: generating a mask for a document image at least inpart via merging regions by pairs of the document image based at leastin part upon minimization of pixel energy, the mask generated such thatan energy variance of a foreground image, a background image and themask is reduced, the energy variance being an estimate of compression;and segmenting the document image into the foreground image and thebackground image, the foreground image and the background image being anon-binary format.
 50. The method of claim 49, further comprising:encoding the mask to generate a mask bitstream; encoding the foregroundimage to generate a foreground bitstream; encoding the background imageto generate a background bitstream; and combining the mask bitstream,the foreground bitstream and the background bitstream into a combinedbitstream.
 51. The method of claim 49, fbrther comprising filling don'tcare regions of the foreground image and the background image with don'tcare pixels.
 52. The method of claim 49, further comprising identifyinglayout information from the mask.
 53. The method of claim 49, furthercomprising removing noise from the mask.
 54. The method of claim 49,segmenting the document image further comprising extending theforeground image and the background image a number of pixels.
 55. Amethod of decoding a document comprising: receiving a bitstream;separating the bitstream into a mask bitstream, a foreground bitstreamand a background bitstream; decoding the foreground bitstream into aforeground image; decoding the background bitstream into a backgroundimage; decoding the mask bitstream into a mask, the mask generated suchthat an energy variance of the foreground image, the background imageand the mask is reduced, the energy variance being an estimate ofcompression; and combining the foreground image and the background imageaccording to the mask into a reconstructed document image.
 56. Themethod of claim 55, further comprising providing display properties. 57.The method of claim 56, the display properties being provided by a user.58. The method of claim 56, the display properties being dynamicallydetermined.
 59. The method of claim 56, the display properties beingpredetermined.
 60. The method of claim 56, further comprising modifyingthe foreground image, the background image and the mask according to thedisplay properties.
 61. The method of claim 55, further comprisingconverting a colorspace of the reconstructed document image.
 62. Acomputer readable medium having computer usable components for adocument encoding system, comprising: a colorspace converter componenttat converts a colorspace of a document image; a mask separatorcomponent that receives the document image from the colorspace convertercomponent and generates a mask for the document image at least in partvia merging regions by pairs of the document image based at least inpart upon minimization of pixel energy; and a segmenter component thatreceives the document image from the colorspace converter and segmentsthe document image into a foreground image and a background imageaccording to the mask, the mask generated such that an energy varianceof the foreground image, the background image and the mask is reduced,the energy variance being an estimate of compression.
 63. The computerreadable medium of claim 62, further comprising: a mask processorcomponent that receives the mask from the mask separator component andprocesses the mask; a clustering component that receives the mask fromthe mask processor component and identifies clusters in the mask; and alayout component that receives the mask from the clustering componentand identifies layout information from the mask.
 64. The computerreadable medium of claim 63, further comprising: a mask encodercomponent that receives the mask from the layout component and encodesthe mask into a mask bitstream; a foreground encoder component thatencodes the foreground image into a foreground bitstream; a backgroundencoder component that receives the background image and encodes thebackground image into a background bitstream; and a combiner componentthat combines the mask bitstream, the foreground bitstream and thebackground bitstream into a combined bitstream.
 65. An encoding systemcomprising: means for generating a mask for a document image at least inpart via merging regions by pairs of the document image based at leastin part upon minimization of pixel energy; means for segmenting thedocument image into a foreground image and a background image; means forencoding the mask into a mask bitstream, the mask generated such that anenergy variance of the foreground image, the background image and themask is reduced, the energy variance being an estimate of compression;means for encoding the foreground image into a foreground bitstream;means for encoding the background image into a background bitstream; andmeans for combining the mask bitstream, the foreground bitstream and thebackground bitstream into a combined bitstream.